To understand what a Light Field Volume is, it’s important
that we first grasp the concept that a captured Light Field consists of rays of
light traveling in every direction, with their brightness, color, and their
path, which is the direction and position of those rays. To learn more about
what is a Light Field, click here to read our previous blog post on the
When light illuminates a scene, light rays bounce off
objects in the scene, reflecting in every direction, mixing colors of each surface
they’ve hit. Rays bounce off surfaces in their path, eventually dissipating
their energy as the light is absorbed gradually with each bounce. Some of those
rays are occluded (blocked) to create shadows while other rays bounce at
different intensities appearing to viewers as reflections or highlights.
In our earlier Light Field blog, we described how a Light
Field can be captured using an array of multiple cameras as well as a plenoptic
device like the Lytro ILLUM, with an array of microlenses placed across its
sensor. The core principle in both cases is that the Light Field capture system
needs to be able to record the path of light rays from multiple viewpoints.
With our new Lytro Immerge Light Field camera for Virtual
Reality, the light rays’ path is captured via a densely packed spherical array
of proprietary camera hardware and technology. In this configuration a
sufficient set of Light Field data is captured from light rays that intersect
the camera’s surface. With that captured Light Field data, the Lytro Immerge
system mathematically reconstructs a spherical Light Field Volume, which is
roughly the same physical dimension as the camera.
From any location within that spherical Light Field Volume,
every viewpoint of the surrounding 360 degree scene can be recreated virtually,
from the furthest left to right, top to bottom and front to back, in every
direction and angle. The Lytro Immerge Player uses that Light Field Volume data
to reproject captured light rays and virtually render the scene, with the rays’
color, brightness and path. The Lytro Immerge Player automatically performs
this critical last step, reconstructing the correct flow of light in the scene to
deliver full parallax movement, natural reflections and shadows, all without
any stitching artifacts.
When viewed through a VR Head Mounted Display placed virtually
inside the captured Light Field Volume via the Lytro Immerge Player, the reconstructed
scene is displayed as perfect left/right stereo views throughout the entire
volume, using any interpupilary distance (IPD) necessary for optimal viewing
experience (distance between the eyes of the viewer). When viewing a Light
Field Volume from within it, the viewer is able to enjoy a truly lifelike,
immersive sensation of virtual presence with full Six Degrees of Freedom and
uncompromised stereo perception at any IPD. Moving forward, side to side or
twisting one’s head, the scene has the same natural parallax as if the viewer
was looking at the original scene. Objects closer to your point of view shift
naturally as you move side to side or back and forth.
In comparison, with typical stereo spherical 2D capture rigs
(a cluster of 2D cameras arranged as a ring or sphere), the individual videos
from each camera must be laboriously stitched into a pair of spherical 2D video
streams. These two streams are recorded at a best-guess IPD, pre-determined and
locked in space when the distance between each pair of cameras was set. As a
result, the stereo depth perception in the VR video is only correct in a single
direction and the stereo effect degrades as the viewer’s head turns.
Conversely, as you look around within the Light Field Volume
using the Lytro Immerge Player, the reprojected scene responds interactively to
your head’s motion, in real time, to precisely match your point of view. Move closer
to the edge of the volume and a smaller area of light ray data is used. Pull your
head back and more light ray data is included in the reconstruction. As you pan
and twist, the player reprojects the correct Light Field rays for your exact viewpoint.
The size and shape of the capture rig directly defines the area of physical
movement a viewer can move around in when exploring a reconstructed Light Field
Volume. In the case of the Lytro Immerge, the navigable Light Field Volume is
slightly smaller than the actual camera size.
Lytro Immerge is designed to be configurable so it can meet
the requirements of any production (the scene being recorded). In the case of a
360 degree scene, the light ray information from every direction is captured in
the Light Field Volume. But other scenes may only need 180 degree views, or
even planar views, and the Lytro Immerge can be configured accordingly to
capture a Light Field Volume that suits the needs of each story. The Light
Field Volume can even be combined with CG elements to produce effects such as a
musical performance on a 3D stage, or create a live-action backdrop to CG
characters. Content creators can also merge captured volumes in post-production
to create larger or differently shaped virtual scenes. In future blog posts in
this series, we’ll cover how various shaped Light Field Volumes and compositing
techniques can be play a part in cinematic live-action content creation for VR.