Volumetric video: a state of play round-table

Feature image: The volumetric capture stage at Avatar Dimension in northern Virginia is the first to utilize IO Industries Volucams.

Leading companies in volumetric video share what they’re doing in this space, and their thoughts on the future of the industry.

Volumetric video is an exciting and still-developing technology. Being such a ‘new’ area also means that the applications for volumetric video are continually being refined—they include holograms, digital humans, gaming, training, brand experiences, metaverse experiences and much more.

A key part of volumetric video, of course, is the capture stage, and there are now several volumetric capture studios established around the world doing this work. In addition, a number of companies in the volumetric video application space have sprung up, offering specific workflow tools to manage and craft experiences with the often heavy volumetric capture data.

befores & afters asked a range of volumetric video outfits—from capture studios to those making workflow tools—about the work they’re doing right now in the field, and where they see it all heading in the near future. Follow along for this special round-table insight, brought to you by IO Industries.

Quick company outlines

Ben Schwartz (technical director, Avatar Dimension): We’re a Microsoft Mixed Reality Capture Studio located just outside of Washington, DC. We work with federal and defense agencies, helping to build immersive training and simulation programs. We view training as the killer app for virtual/augmented reality, and volumetric video is the best method for inserting realistic human presence into training modules. But we also provide volumetric video capture for enterprise, entertainment, historical preservation, and more.

Atsushi Katsumoto (director at Softbank/Realize Mobile Communications (RMC)): We are running three 8i Stages in Tokyo, Japan, for capturing and real-time streaming ​volumetric video, based on the strategic partnership between 8i and RMC to expand the market of volumetric video in Japan. We believe that the volumetric video streaming is the future of the telecommunication.

Jack Tipton (founder, GenVR): We create realistic VR sports training simulations and post-match analysis through the use of volumetric video and free viewpoint video technologies. We have also set up our own studio in Manchester with portable capabilities.

Victor Pardinho (co-founder & CEO, Sense of Space): Sense of Space was born out of years of R&D with volumetric video, photogrammetry and 3D data. The company started by helping creators, studios and other companies on how to handle the complex volumetric data, that can take huge file sizes and be almost impossible to work with in a traditional VFX workflow.

Piotr Uzarowicz (head of partnerships & marketing, Arcturus): Arcturus allows creators, artists and agencies to work with volumetric data from any source so they can edit, compress and distribute volumetric videos to a platform of their choosing.

Tom Emrich (VP of product, 8th Wall): 8th Wall is the leading WebAR development platform. We equip developers, agencies and brands with the tools they need to create World Effects, Face Effects and Image Target augmented reality experiences in the browser. 8th Wall WebAR experiences require no app to download and support nearly 3.5 billion smartphones across iOS and Android. 8th Wall’s Engine also features Metaversal Deployment which enables World Effects to work across smartphones, computers and both AR and VR headsets, significantly increasing the reach for developers without increasing the development time.

Key tech

Atsushi Katsumoto (Softbank/RMC): The 8i Stages have multiple types depending on your needs. There is one studio with 30 4K cameras and two studios with 30 2K cameras. It consists of IOI Victorem cameras, ARRI lightings, and 8i software that automatically syncs all the cameras. The 8i software is installed on 10 recording machines and 1 master machine to capture and stream volumetric video. You can also add rendering farms for a faster rendering depending on your needs. In our case, we have built nearly 100 rendering machines to handle a large number of shots.

Ben Schwartz (Avatar Dimension): Our studio is comprised of 70 IO Industries Volucams. The Volucam is the first camera purpose-built for volumetric capture, and our studio is the first to utilize it. We’ve built out what I feel is a clever fiber-based infrastructure for the studio. All Volucams are cabled with a single strand of 10GbE fiber. That fiber connection handles both interface and data transfer. All of our cameras are connected to a series of cassettes at the top of every tower that multiplexes all the individual strands of fiber into two multi-strand fiber cables that we run from the studio back out to a series of switches and on to storage/processing. So rather than having to run 70 individual fiber cables between the studio and the switches, that number is reduced down to 18.

Jack Tipton (GenVR): We are a small, independent studio with 2000 sq ft of space of which just over 1,000 is dedicated to volumetric capture. From a hardware perspective, we use IO Industries Volucam cameras, of which we have 16 capturing at 4K resolution at 120fps. We then make use of 8mm machine vision lenses from FujiFilm to maximise our capture space whilst keeping edge distortion to a minimum. In terms of lighting, we have 12 Rayzr MC400 Max light panels. We use IO Industries StudioCap software to control camera settings and capture video, of which we then transfer to Volograms software for asset reconstruction.

Victor Pardinho (Sense of Space): Our first product, Sense XR Studio, is a volumetric video content creation tool for AR, VR and Web platforms. We think Sense of Space has one of the best formats in the market, cloud-based compression and streaming system for volumetric video and also photogrammetry or any kind of highly-detailed 3D geometries, holograms and models.

Piotr Uzarowicz (Arcturus): Volumetric data tends to be large and prone to artifacts due to the nature of the capture technologies. Our HoloSuite tools allow experienced and inexperienced users to perform mesh clean-up, decimation and adaptive compression in a way that the data can be efficiently interacted with on any platform, and under any bandwidth constraints.

Tom Emrich (8th Wall): 8th Wall has partnered with a number of volumetric video solutions to make hologram content available within 8th Wall projects. We work with these partners to integrate volumetric video playback within WebAR including equipping developers with sample projects to help them get started creating WebAR hologram projects. We’ve also provided resources to help developers add interactive elements to the scene such as playback systems, or being able to pick up the hologram and move the hologram in space.

Recent collaborations and project highlights

Victor Pardinho (Sense of Space): We work with several companies in the field, especially the Microsoft Mixed Reality Studios such as Metastage and Avatar Dimension. 4D Views Studios, like 4DR Studio in Eindhoven. TetaVi, Mantis Vision and many other studios and creators around the world.

Atsushi Katsumoto (Softbank/RMC): The 8i Stages have been used in many commercial projects such as AR experiences for 5G promotion, VFX at fashion shows, 3D stamps on SNS, and live conference at SIGGRAPH Asia. It has been an honor collaborating with a wide variety of top performers, artists, singers, comedians, athletes and dancers.

Tom Emrich (8th Wall): 8th Wall has partnered with a number of volumetric video companies including Mixed Reality Capture Studios (MRCS), 8i, 4Dviews, Arcturus, and Tetavi. We’ve seen a number of brands make use of volumetric video as part of their WebAR experiences.

Ben Schwartz (Avatar Dimension): We recently wrapped up a series of heritage captures for the Osage Nation, a midwestern Native American tribe here in the US. They sent over a couple of elders to recite a series of ritual prayers in the Osage language. We view heritage preservation as one of the key use cases for volumetric video. We also just delivered a musical performance capture for a country artist named Ryan Sims that’s going to be incorporated into a 2D music video. And we recently completed a PSA for COVID vaccination with Dawn Staley, the head coach of the University of South Carolina women’s basketball team. We captured her and one of her players separately, and then combined the two captures into a single mobile AR volumetric piece using Sense of Space.

Jack Tipton (GenVR): The development of our studio was not initially set up for third party use, it was mainly for our own private use cases but we have made it available for others to use as well through our secondary website volustudio.com. I think given the unique properties of our setup there will be more options for usage by smaller companies who don’t necessarily have the resources to partner with the big organisations, or for increased portability, bringing the studio to the talent. There’s also the option for different types of volumetric capture that previously hasn’t been possible given the size constraints of traditional setups, for example, sports motions.

Piotr Uzarowicz (Arcturus): We have worked with every capture system around the world and most of the volumetric capture studios. We maintain a map of all the volumetric stages around the world. We’ve had a few high profile projects in sports and music recently, including the Madonna holograms that were part of the live TV broadcast of the 2019 Billboard Music Awards, Pose with the Pros with the Dallas Cowboys, and one with K-Pop band BTS with the launch of a Samsung phone. We’ve also been a part of really impressive uses of volumetric video for film and episodics in projects like Hulu’s “Light as a Feather,”and Asad Malik’s “A Jester’s Tale,” which premiered at Sundance, to name a few. We’ve also seen a lot of projects coming out of ecommerce from companies like Anayi, as well as games, telecom and more.

The future of volumetric video

Tom Emrich (8th Wall): The future of volumetric video is bright. I anticipate that we will continue to see more professional capture studios pop up around the globe making it even easier for agencies and brands to create commercial volumetric video. I also anticipate advancements in capture hardware which will enable more portable setups, new features around editing and playback, and a distinct focus on bringing volumetric video into the mainstream whether through studios or by making use of the smartphone camera and LiDAR sensors.

Atsushi Katsumoto (Softbank/RMC): We truly believe that the volumetric video technology is the reinvention of the camera technology. The volumetric video will also be used in every industry just like the camera technology is currently used in all fields like smartphones, films, and autonomous driving. Our vision for it to be the basic infrastructure of communication.

Victor Pardinho (Sense of Space): Volumetric video is still in its experimentation phase, both creatively and technologically. Unfortunately it’s still expensive to make volumetric video captures in a professional studio for most creators and producers out there, but more accessible solutions are already there and coming. Those that use Microsoft Kinect sensors for example such as Depthkit and SOAR, or even using a single phone, such as Volograms, or live-streaming of volumetric data like Condense Reality. When it gets easier and more accessible to capture volumetric videos, we will certainly see more productions and more interesting content being made using the technology.

Jack Tipton (GenVR): I think there are diverging paths with volumetric video, on one hand you have the current setups of high end captures but in controlled, rigid environments, and although more flexible than others, I would include our current setup in that. On the other hand I think we will move towards live event capture using fewer cameras, which is where we are developing our own market offering. I’m not sure the 40-80 camera setups currently trialled in live event capture are viable long-term. I think the setup costs involved in these arrays will always be too high.

Piotr Uzarowicz (Arcturus): The capture of a true-to-life human performance has several obvious applications in any metaverse setting, but we see it going well beyond that. The rise of virtual productions confirms that there’s a lot of room to leverage this human capture technology for virtual environments. Single sensor 3D capture using machine learning is also going to have a huge impact as avatars are going to be a crucial piece in any virtual universe, and the more people that can access and create them, the better.

Ben Schwartz (Avatar Dimension): One thing we think about a lot is volumetric video and its place in the so-called metaverse. Digital humans and avatars in the metaverse so far seem to be almost entirely synthetic and cartoony looking, and I think there is still some degree of uncanniness with even the highest quality avatars. Only volumetric video pushes past the Uncanny Valley into the realm of true photorealism. Now, whether or not there’s a place for photorealistic human presence in the metaverse is an open question. People are often looking to make their avatars into something more than human; not always necessarily inhuman, but let’s say superhuman. And volumetric video is about capturing people as they are. I personally think there will be a definite place for realistic human presence within the metaverse, but obviously we’re still figuring it all out.