To understand what a Light Field Volume is, it’s important that we first grasp the concept that a captured Light Field consists of rays of light traveling in every direction, with their brightness, color, and their path (which is the direction and position of those rays). To learn more about what is a Light Field, click here to read our previous blog post on the subject.
When light illuminates a scene, light rays bounce off objects in the scene, reflecting in every direction, mixing colors of each surface they’ve hit. Rays bounce off surfaces in their path, eventually dissipating their energy as the light is absorbed gradually with each bounce. Some of those rays are occluded (blocked) to create shadows while other rays bounce at different intensities appearing to viewers as reflections or highlights.
In our previous Light Field post, we described how a Light Field can be captured using an array of multiple cameras as well as a plenoptic device like the Lytro ILLUM, with an array of microlenses placed across its sensor. The core principle in both cases is that the Light Field capture system needs to be able to record the path of light rays from multiple viewpoints.
With our new Lytro Immerge Light Field camera for Virtual Reality, the light rays’ path is captured via a densely packed planar array of proprietary camera hardware and technology. In this configuration a sufficient set of Light Field data is captured from light rays that intersect the camera’s surface. With that captured Light Field data, the Lytro Immerge system mathematically reconstructs a spherical Light Field Volume, which is roughly the same physical dimension as the camera.
From any location within that Light Field Volume, every viewpoint of the surrounding 360 degree scene can be recreated virtually, from the furthest left to right, top to bottom and front to back, in every direction and angle. The Lytro Immerge Player uses that Light Field Volume data to reproject captured light rays and virtually render the scene, with the rays’ color, brightness and path. The Lytro Immerge Player automatically performs this critical last step, reconstructing the correct flow of light in the scene to deliver full parallax movement, natural reflections and shadows, all without any stitching artifacts.
When viewed through a VR Head Mounted Display placed virtually inside the captured Light Field Volume via the Lytro Immerge Player, the reconstructed scene is displayed as perfect left/right stereo views throughout the entire volume, using any interpupilary distance (IPD) necessary for optimal viewing experience (distance between the eyes of the viewer). When viewing a Light Field Volume from within it, the viewer is able to enjoy a truly lifelike, immersive sensation of virtual presence with full Six Degrees of Freedom and uncompromised stereo perception at any IPD. Moving forward, side to side or twisting one’s head, the scene has the same natural parallax as if the viewer was looking at the original scene. Objects closer to your point of view shift naturally as you move side to side or back and forth.
In comparison, with typical stereo spherical 2D capture rigs (a cluster of 2D cameras arranged as a ring or sphere), the individual videos from each camera must be laboriously stitched into a pair of left/right spherical 2D video streams. These two streams are recorded at a best-guess IPD, pre-determined and locked in space when the distance between each pair of cameras was set. As a result, the stereo depth perception in the VR video is only correct in limited directions and the stereo effect degrades as the viewer’s eyes are tilted or gaze up/down.
Conversely, as you look around within the Light Field Volume using the Lytro Immerge Player, the reprojected scene responds interactively to your head’s motion, in real time, to precisely match your point of view. Move closer to the edge of the volume and a smaller area of light ray data is used. Pull your head back and more light ray data is included in the reconstruction. As you pan and twist, the player reprojects the correct Light Field rays for your exact viewpoint. The size and shape of the capture rig directly defines the area of physical movement a viewer can move around in when exploring a reconstructed Light Field Volume. In the case of the Lytro Immerge, the navigable Light Field Volume is slightly smaller than the actual camera size.
Lytro Immerge is designed to be configurable so it can meet the requirements of any production (the scene being recorded). In the case of a 360 degree scene, the light ray information from every direction is captured in the Light Field Volume. But other scenes may only need 180 degree views, or even planar views, and the Lytro Immerge can be configured accordingly to capture a Light Field Volume that suits the needs of each story. The Light Field Volume can even be combined with CG elements to produce effects such as a musical performance on a 3D stage, or create a live-action backdrop to CG characters. Content creators can also merge captured volumes in post-production to create larger or differently shaped virtual scenes. In future blog posts in this series, we’ll cover how various shaped Light Field Volumes and compositing techniques can be play a part in cinematic live-action content creation for VR.