The Influence of Visual Cues on Sound Externalization

Background: The externalization of virtual sounds reproduced via binaural headphone-based auralization systems has been reported to be less robust when the listening environment differs from the room in which binaural room impulse responses (BRIRs) were recorded. It has been debated whether this is due to incongruent auditory cues between the recording and playback room during sound reproduction or to an expectation effect from the visual impression of the room.

This study investigated the influence of a priori acoustic and visual knowledge of the playback room on sound externalization.Methods: Eighteen naïve listeners rated the externalization of virtual stimuli in terms of perceived distance, azimuthal localization, and compactness in three rooms: 1) a standard IEC listening room, 2) a small reverberant room, and 3) a large dry room.

Before testing, individual BRIRs were recorded in room 1 while listeners wore both earplugs and blindfolds. Half of the listeners were then blindfolded during testing but were provided auditory awareness of the room via a controlled noise source (condition A). The other half could see the room but were shielded from room-related acoustic input and tested without the controlled noise source (condition V).

All listeners were also tested with all cues available (condition AV). Seven azimuthal source positions were reproduced, with loudspeakers visible at four azimuthal positions. Results: In condition AV, the auditory images were perceived closer to the listener in rooms 2 and 3 than in room 1, with a larger effect in the reverberant than in the dry environment.

In room 2, the perceived distance of the virtual sounds was more accurate in condition V than in conditions A and AV, where it was reduced. In room 3, differences in distance judgments between A, V, and AV conditions were much less pronounced. In contrast to distance, localization and compactness judgments were largely room independent, although localization judgments were less accurate and compactness ratings less consistent in conditions V and A than in condition VA.

Conclusion: A mismatch between recording and playback room was found to be detrimental to virtual sound externalization. The auditory modality governed externalization in terms of perceived distance when cues from the recording and playback room were incongruent, whereby the auditory impression of the room was more critical the more reverberant the listening environment was.

While the visual impression of the playback room did not affect perceived distance, visual cues helped resolve localization ambiguities and improved compactness perception.