The heartbeat of multimodal creativity: A pilot heart rate study on objectivity and subjectivity in audio description

Abstract

The appropriateness of subjectivity or objectivity in audio description (AD) is an ongoing debate where the professional and the academic perspectives seem to collide. Whereas some professional guidelines support the view that an objective AD is the advisable option, some scholars have pointed out the advantages of more creative alternatives. Meanwhile, part of AD research is making inroads into experimental methodologies: AD and eye-tracking, text-to-speech AD, the automatic generation of AD, and physiological measurements applied to AD, such as heart rate, cortisol levels, or skin conductance response, among others. In this vein, our proposal presents an experimental pilot study using a heart rate sensor to measure partially sighted participants’ response to different combinations of subjective and objective multimodal components within the AD script. To do so, objective and subjective multimodal components from an AD clip from Netflix’s Money Heist were analysed. Afterwards, objective components were turned subjective (and the other way around), which resulted in two versions of the clip in subjective-objective terms: the original and the creative one. Both versions were recorded by a professional studio, participants were randomly assigned one of the clips, and they orally completed a demographic survey, a presence test, and a comprehension test followed by a semi-structured interview. Moreover, the UPTIVO Belt-D heart rate sensor was used to measure beats per minute, average heart rate, maximum heart rate, minimum heart rate, etc. Preliminary results were triangulated with the tests and the interview to obtain empirical data about the combination of subjective and objective multimodal components in AD that induces the highest heart rate and presence levels, enables better comprehension and is best regarded by users. Future research following this methodological design and involving an appropriate number of participants could provide empirical evidence about the immersive combination of objectivity and subjectivity in AD to be considered by guidelines.

Keywords
multimodality; audio description; creativity; heart rate; experimental research