Introducing the IR & Inciprocal Digital Double Pipeline (Part 1)
We’ve been working closely with Inciprocal for the last few years, designing and specifying out a capture pipeline utilizing our AeonX: S0L scanning rig to fit within the Inciprocal software pipeline. Our scanning and digital double sculpting work was recently featured in Unity’s Enemies demo and the ZIVA acquisition presentation.
Today’s inspiration:
[soundcloud url=”https://api.soundcloud.com/tracks/1287497917″ params=”” width=” 100%” height=”166″ iframe=”true” /]
Our R&D Results
You can see an early example of our R&D digital double research. We utilized IR’s captured data combined with the Inciprocal full Disney PBR texture stack. We took these processed results added some extra modifications, like extra sculpting and texture work. We then re-built our LED rig virtually and visualized the results inside Marmoset Toolbag 4’s raytraced virtual environment.
Split images. Digital Double on the left. Real-World subject on the right.
What’s Required?
The more data that can be fed into the Inciprocal software pipeline the better and with as minimal subject movement as possible. IR’s machine vision cameras are essential for this combined with IR’s IDA-Tronic LED system which is lightning fast and highly programmable. Other LED rigs on the market are sometimes limited by the amount of programmable lighting states that they can store. The IDA-Tronic system has the ability to store over 120 different lighting states at any one time, plus as many OLAT states as you require.
This novel capture and processing pipeline is not using a spherical gradient or binary gradient method.
We decided on 36x pairs of cross-polarized and parallel-polarized TRILAT lighting states to be fed into the Inciprocal pipeline. That’s a total of 76x lighting states, plus 2x global illumination states for a total of 78x captures, as well as an additional 156x OLAT shots at the end of the sequence for our main reference cameras. This is required for additional ground truth “unfiltered” reference.
We use a cluster of triplets (TRILAT) to output as much light as possible. We capture at millisecond speeds.
System Specification
We’ve designed and built a hybrid custom photometric / motion scanning system from the ground up, utilizing 59x 12MP machine vision cameras from Ximea. We use polarizing filters on 56x of our main cameras and no filters on the 3x main cameras cam57, cam58 & cam59.
TRILAT Examples (cam01, cam19 & cam35)
Speed is essential!
Machine Vision Cameras & LED’s
We utilize our own triggering system to sync the LED’s and cameras together, as well as a custom software solution called “AeonX” to interface with the cameras.
The LED system comprises 312x LED’s with 3 different controllable sectors (Sector0, Sector1 and Sector2). The whole system is extremely programmable. We utilize polarizing filters on some of the 3 sectors and on some of the cameras.
In a previous Blog post, we showcased our SDL pipeline. When operating in this new “TRILAT Inciprocal” mode we can trigger the S0L system at 62.5 fps. This allows us to capture 78x different lighting states (or scans!) in 1.248 of a second! Compared to our older DSLR system which would trigger at between 3-5fps which was MUCH slower and not reliable, or also in comparison to mirror less Sony cameras that trigger at a slower pace of about 20fps.
Triggering Dozens & Dozens of Cameras in a Split Second
We’re able to trigger all of our 59x Ximea machine vision cameras at blazingly fast speeds to acquire thousands of images in under 1.248 seconds. This data is then streamed directly to disk, instantly, for fast playback and observation.
In this mode, to produce a single scan we would typically acquire 4,602 images, at about 55GB’s per expression. Below you can see a subset of 56x cameras in a row, representing each lighting state.
We can also accompany this TRILAT Inciprocal set with a post OLAT spiral set of another 156x “no-filter” illuminations, triggering just the 3x “no-filter” reference cameras. That’s another 468x images.
Why do we need all this data? 1) Because we can! 2) Having as much real-world reference data as possible REALLY helps with the Digital Double look dev validation stage. Whether it’s to check skin reflectance, refractive caustics in the subjects eyes, the anisotropic properties of hair strands, vellus back scattering reference, subsurface scattering reference. The list goes on. The data is invaluable.
Validation Reference Images
We can then re-build our TRILAT images from the OLAT spiral set from the perspective of cam57, cam58 & cam59 (no-filters) to produce a pure unfiltered reference set. The OLAT set and the re-built TRILAT set can then be used as the true render validation for the digital double aspect of the pipeline once Inciprocal has processed and delivered the TRILAT data back to us. This RAW scan will act as the base template for Inciprocal to process with.
Inciprocal Processing Pipeline
Once the scanning session is complete we will build the base photogrammetry scan and provide the RAW scan and calibration data from REALITY CAPTURE to Inciprocal ready for their part of the differential rendering pipeline. After processing at Inciprocal they will then deliver the albedo, reflection, roughness, world space, tangent space and displacement texture maps. We can then use this data to render match to IR’s reference captures and camera alignment from REALITY CAPTURE.
The beauty of the Inciprocal pipeline is the accuracy and detail of the output maps. A full Disney PBR texture stack is reconstructed that usually takes around 40-50 minutes on a single @NVidia RTX 3080. The data is plug and play and acts as great base template to work from to further enhance (if required).
We’re working on optimizing the amount of data captured to be able to support expressions. This is further R&D work to be carried out.
We’re really excited about the potential of our new AeonX: S0L system and the Inciprocal pipeline. We’re looking forward to working with clients on new and exciting projects and pushing the boundaries of what’s possible with 3D scanning and facial capture technology.
Look out for Part 2 of our Digital Double Pipeline..
Thanks for reading
