How We Used VR to Explore What Music Feels Like to a Deaf Person

Seven years ago, when Rachel Kolb was 20, her friends pitched in to help her learn how to hear music. She was born profoundly deaf and had recently received a cochlear implant to give her partial hearing. “They were so gracious — they made a playlist and annotated it: At this time in the song, it’s this instrument coming in,” she recalled. “So I learned to recognize, oh, that’s a piano, because my friend wrote down, ‘At 35 seconds the piano starts to play.’ ”

Rachel is a former Rhodes scholar and current doctoral student whom I met through Peter Catapano, the editor of the Disability series, on the Opinion desk. I work as a senior producer in the VR department at The Times, creating both long- and short-form virtual reality videos. Part of my job is constantly looking for stories that will fit our uniquely experiential medium.

Peter introduced me to Rachel after she submitted an essay to him about her experiences of music both before and after receiving the implant. She described music as tactile and visual — not something that you just hear. We thought her story was a great match for the immersive treatment that virtual reality provides. We started to adapt Rachel’s article into a storyboard and quickly settled on a VR piece that would be a mix of animation and live-action, with narration from Rachel.

Continue reading the full article at The New York Times:
About Lytro 30 Articles
Lytro is building the world’s most powerful Light Field imaging platform enabling artists, scientists and innovators to pursue their goals with an unprecedented level of freedom and control.

Be the first to comment

Leave a Reply

Your email address will not be published.