Understanding the Light Field viewing volume, it’s practical applications in VR and how viewers interact with it are core challenges at Lytro. It forced us to think in new ways and find solutions that work in conjunction with the broader ecosystem of post-production tools and storytelling conventions.
In a previous blog post we talked about the concept of the Light Field volume.
We often refered to this Light Field volume as a sphere that surrounds the viewer’s head. In this blog post we’ll explore the various considerations that come into play when shaping the view volume and how it affects the resulting user experience.
First, let’s understand what is the “natural” shape of the view volume. The view volume’s shape depends on the shape of the device that captured the footage/rays of light. For example, if you have a perfect dense sphere of cameras, the view volume would also look like a smaller sphere in the center of the capture sphere. How much smaller? That depends on the field of view of the lenses involved. For example, with 90 degrees lenses, the view volume of a capture sphere would look like this:
Cross section of a capture sphere (in gray) surrounded by cameras with 90 degrees lenses,
forming a smaller view volume (in orange) in the same shape as the capture sphere.
If you have a planar capture configuration, the view volume shape starts being less well defined. The resulting view volume shape is a collection of overlapping triangles that form some sort of a star configuration (exact shape is determined by the rotation steps and field of view of the lenses).
Why does the shape matter?
The shape of the viewing volume matters due to several important considerations:
- It defines the range of motion (space) the viewer would eventually have
- It defines the continuity of motion inside of the view volume
- It affects the production cost
- It affects the creative choices and data size of the final output
1. Range of Motion
Range of motion is simply the volume of space that is available for the viewer to move in. The larger the volume, the more the viewer’s head can travel within the Light Field, the more parallax is available, the more details that can be discovered (revealing occluded information). All things being equal, most content creators would always prefer to have as large of a volume as possible – bigger is better!
The balance to volume size is data size. As described in the “Outward vs. Inward Capture” blog post, the data size in a volume grows to the power of 3 and that quickly gets very large. Depending on the use-case, and method of delivery, this can become unwieldy and defeats the purpose of a great viewer experience.
2. Continuity of Motion
The continuity of motion consideration within a viewing volume has to do with the geometrical shape of that volume and how the player application handles “Out of Bound” (OOB) scenario. When the viewer find him/herself in a partially samples point of the Light Field, unpredictable results might occur. The required rays of light might be available or not. The resulting distortions or artifacts might be noticeable, or not. The player might be able to reproject other rays of light into the missing pieces, or not – it’s simply not predictable and therefore we always try to avoid it.
As an example of discontinuity within a Light Field, let’s imagine shape with a missing triangle (represented by red lines below). Since the viewer can freely move their head in real-world coordinates, they can easily find themselves in a zone of discontinuity where the Light Field is not fully sampled.
Viewer moving their head from point A to point B in a curve find themselves
inadvertently outside (highlighted in red) of the viewing volume (orange).
3. Production Cost
As stated before, the larger the volume, the more data you need to manage from capture to post-production all the way to final delivery. Due to current technological/IT constraints (storage, data throughout, backups, etc.), cost is often influenced by data size and content producers have to ask themselves – is that enough volume or possibly too much?
For post-production, the more data you manage, the more touchups and cleanups you have to evaluate and that can quickly add-up. When performing post-production operations on a volume of space, you have to constantly deal with visual information that not only changes over time (temporally) but also based on spatial point of view (result of parallax). An actor’s feet that seem invisible from one angle can suddenly pop into frame from another angle and require some adjustment. An actor’s face that is perfectly lit from the front might be underexposed from the side or might reveal the cable of the spot-mic.
Storage and rendering costs can also grow in the same direction. There are many techniques and best practices for mitigating cost increases but the general rule is: more volume = more cost. The Light Field volume captured by Lytro Immerge is carefully selected to strike a balance between view volume and overall production cost.
4. Final Output
Most importantly, the shape of the viewing volume has a direct impact on artistic choices and storytelling. Assuming that content creators are trying to balance all of these considerations and produce the best VR experience for their project, shaping this viewing volume becomes a critical creative choice. For example, imagine a performer in a corner of a room. The room is providing critical atmosphere for the performance but the viewer is expected to spend 90% of their time looking at the performer. Investing production cost and data size in enabling range of motion for viewing the room might be a misguided choice vs. enabling greater range of motion near the performer – getting closer, looking from the sides or from above…
Another example is practical set. A director might choose to give a nice range of motion for the audience in front of a stage but trim that motion on the edges, just before the set ends and the back-stage starts becoming visible. Or the opposite, expose a surprise if the viewer is looking at the back-stage.
Content creators have to balance many considerations and find the right solution for each situation/need. It is recommended to think about the most demanding situation that you have (the largest viewing volume that you think you need) make that into your “master volume” and customize multiple viewing volumes as a subsets, each addressing a different need. The largest viewing volume could be for an in-venue experience (where file size is not an issue). A slightly trimmed down version for a large download and a small viewing volume for a smaller download version.
To address all of the considerations and challenges mentioned above, Lytro has been developing a wide range of solutions, including:
- Plugins that extend the capabilities of existing post-production tools to accommodate Light Field and three dimensional points of view
- Cloud processing infrastructure that can help optimize time and cost
- Quality control tools for tracking every ray of light, where it originated from and how it affects the viewer’s experience
- Tools for tracking the range of motion viewers require while viewing different VR pieces so the viewing volume can be optimized
As it pertains to tracking viewers motion in headset, we are also interested in different viewing experiences from seated positions to standing in place or walking around a large, room-scale, Light Field. The goal is to develop tools and best practices that help guide content creators in the pursuit of the optimal Light Field viewing volume for each piece of content and use.
Viewer being tracked for range of motion while viewing Lytro’s “Moon” VR piece in a seated position.
Data is represented as a heat map where red = frequent and green = less frequent visit
For live action Light Field content, we think of a sphere as that natural shape that we need to capture. Once obtained, we treat this large sphere-shaped view volume as the ultimate “master” and all optimizations are a subset of that master, chosen based on creative choices, production cost and delivery needs.
Various shapes of viewing volume around a viewer’s head. The larger gray sphere represents the “master” view volume and the orange shape is the optimized view volume. Each piece of content might benefit from multiple different shapes to meet different objectives.
In developing Light Field solutions, we often find ourselves looking to solve challenges in an uncharted territory. We are constantly iterating and experimenting with different view volume shapes, sizes and characteristics. In our persistence, we strive to give content creators the widest set of creative tools and choices with which to tell their stories, and look forward to seeing how they will put it all to great use.