Introducing IR’s AeonX: S0L Photometric Pipeline

For the last few years, we’ve been busy re-designing our photometric and motion scanning system as well as our pipeline. We’ve not posted much on social media or on our website simply because our systems have been in flux and we’ve been super busy with client work. We’re always pushing and always trying to fine-tune and tweak our systems and the scanning service we provide.

[soundcloud url=”https://api.soundcloud.com/tracks/886560637″ params=”” width=” 100%” height=”166″ iframe=”true” /]

System Specification

We’ve designed and built a hybrid custom photometric / motion scanning system from the ground up, utilizing 56x 12MP machine vision cameras from Ximea and 3x 24MP Volucam cameras from IOI Industries used purely for reference capture. The system operates in 2 different modes shown below.

“Motion scanning” (60fps)

Photometric/SDL “Sequenced Directional Lighting” (62.5fps)

We utilize our own triggering system and software solution to control and operate the cameras. Our camera system is complemented by the IDA-Tronic LED system

The LED system comprises 156x LED’s with 3 different controllable sectors (Sector0, Sector1 and Sector2) and the whole system is extremely programmable. We utilize polarizing filters on some of the 3 sectors and on some of the cameras.

When operating in SDL mode we can trigger the system at 62.5 fps and this allows us to capture 20 different lighting states (or scans!) in 0.288 of a second! Compared to our older DSLR system which would trigger at between 3-5fps which was MUCH slower and not reliable, or also in comparison to mirror less Sony cameras that trigger at a slow pace of about 20fps.

Building & Researching the System

When we first started building photogrammetry systems back in 2009, the idea of syncing multiple cameras once, reliably, was a massive technical challenge. Now we’re able to do it multiple dozens of times a second, without fail.

Our new system is lightning fast and allows for very rapid image acquisition straight to disk on an operating system, eliminating the issue of subject movement and having to “stitch images” together or use optical flow. Also, because we capture at such high frame rates, we can pulse our LEDs very rapidly in sync with the cameras, which removes the issue of motion blur and also prevents blinding the talent with bright flashlights, like our older single shot “DSLR photogrammetry systems” tended to do. Also, as a bonus.. no exploding flash bulbs!

It’s quite difficult to express and visualize how much data we are able to acquire. So the below images are just a small subset of what’s captured.

Another added bonus of the system, for certain expressions, we can also add an additional set of lighting captures called a “Spiral” set to capture all the required OLAT shots of the subjects from each LED. This will add an additional 156 lighting states, which takes around 2 seconds for a total of 176 images (or scans!) per camera!

Camera Arrangement (BugEye display!)

All of the 20 image sets below were captured in about 0.288 of a second. To the human eye, it looks like a single flash.

To give you an idea of the amount of image data captured from a single photometric / SDL / OLAT scan = 176 images per camera x 59 cameras = 128GB of data x the number of expressions a project might require and that’s just for the RAW image data. That’s 128GB of raw image data for just one expression.

Reference Camera Closeup

You can see below a set of captures from our 3 main reference cameras used for render matching later in the pipeline. We make sure to calibrate each cameras exposure and white balance correctly to an xrite chart.

This pipeline is then carried through to the FACS part of the capture process, so the above datasets are acquired for each subjects expression scan. We can scan as many expressions as a client requires.

Expression Data Sets

We can capture a wealth of data rapidly. These data sets are ideal for machine learning applications and skin reflectance analysis. Each one of these expression’s scans (combined data sets) took less than 0.28 of a second to capture.

We’re really excited about the potential of our new system and the quality of data that we can produce. We’re looking forward to working with clients on new and exciting projects and pushing the boundaries of what’s possible with 3D scanning and facial capture technology.

Edit 2022/12/22
Part of IR’s photometric process is based on the incredible research by Paul Debevec and the team at USC & ICT. In combination with our own software and hardware processes we use these licensed and patented technologies (license application pending 2023).

In the meantime we are commercially deploying a “Trilat” capture process in combination with image processing @ inciprocal.com

We hope to post more information related to the “Trilat” process soon.

Thanks for reading