Blog

Discover what makes us unique
← Back
UE4 Skin Shader and Virtual Character Rendering

UE4 Skin Shader and Virtual Character Rendering

March 28, 2016 - Blog - - 25 responses

I wanted to make a post to share my findings with regards to skin shading and rendering realistic characters in Unreal Engine 4.11. We need some help from Epic on the core skin shader implementation..

Excuse the long post but it’s all related to an attempt at shading and rendering realistic characters in UE4.

Today’s inspiration:


.

Epic we need your help 🙂

Following on from http://ir-ltd.net/next-gen-photometric-scanning/ This type of acquired accurate data is ideal for cross checking with PBR shaders in UE4 or offline renderers. IR’s rig captures the multi lighting data during the scanning session, not after, for solid alignment and comparisons. It’s a 1:1 match to the scan data itself. All this data can be mapped back onto the scan set. Including the backscatter images, which tells us information about translucent areas of skin. Or the specular separated data which can reveal the more oily areas of skin.

Output Data from IR’s Rig

Photometric Normals and Multi Lighting Data

ir-output-w

Flash Captures

nick-example-multi-shots-crop

.
Custom UE4 Skin Shader

We’ve been working with UE4’s awesome new shaders and wanted to share some example results and get an open dialogue going. The results are by no means perfect due to lack of control and real-time rendering limitations. The custom shader changes were made by Charles Greivelding https://www.artstation.com/artist/charlesgreivelding

(missing back)

multi-light-mix-w multi-light-double-w multi-light-left-w multi-light-bottom-w multi-lighttop-w multi-backlit-w multi-light-front-w

With real-time rendering we miss many features that offline renderers have, like multiple GI bounce, fine and deep subsurface scattering, true ray traced shadows (soft penumbra) and ray traced reflections.

When trying to render in real-time applications there are many obstacles to over come when trying to match against offline renderers like Vray or Arnold. We have to sacrifice a certain amount of realism but real-time renderers like UE4 and Unity are getting better and better.

We are continually inspired by the work of Jorge Jimenez who IR worked with back in 2013 on “Next Generation Character Rendering” for Activision.

Jorge’s work has inspired many studios since, like Epic, who have integrated his techniques into UE4, like his skin shading solution and eye shading tricks. Which were recently seen in Ninja Theory’s awesome Hellblade demo, featuring Senua.

We need some further guidance from Epic on shader integration and possible core updates. As we have found some limitations with the skin shader, which can hinder realism, some of Jorge’s features are missing. We hope to make a case that might inspire Epic further (or not 🙂

Arnold uses The GGX distribution which is new to Arnold 4.2.12 and generally matches real-world materials better than the default Beckmann distribution. With access to 2x GGX specular controls. Epic uses only 1 GGX specular control.

Some missing controls:

Here is an example of how Maya and Arnold render skin

Notice the tight 1st specular lobe we see on the micro details, combined with the larger rougher bluiesh 2nd specular lobe.

(rendered with Maya and Arnold using http://www.anderslanglands.com/ alshader )

Image converted using ifftoany

Image converted using ifftoany

1) Not having access to 2x specular lobes (one white, tight and fine and one blue, large and rough) which is very important for skin shading on micro details
2) No built in back scattering (deep SSS) and multi coloured SSS, for thinner areas of skin like in the ears. For these changes to happen, UE4 needs a core update to unlock these features?
3) Normal maps are also locked at 4K resolution. We can get in 8K colour but not 8K normal maps, which are essential for details. No access to 16k micro normal application.
4) We currently cannot get the Eye shader iris depth to work and the reflections cause darkening errors when compared to default lit or subsurface.  Either user error (see node network) or bug
5) We have a wet blur layer (forward hack) but have no access to screen space shadows (light leak at corners of eyes) and no way to control depth blur getting larger at further distances from subject

We’ve since been able to hack together a custom version of UE4.11 with 2x specular lobe control (bad hack), back scattering deep SSS and multiple micro tile blending (which we are still working on)

This means running a customized (50gb!!!) 4.112 version of UE4 to accomplish this. Hopefully this post might inspire some of the devs at Epic to look into further upgrading the skin and to hopefully look further into 2 specular lobe controls. With that we can get much better specular shine on skin, as at the moment it either looks to dry or too wet and we have to battle with the controls.

Backscattering

Example Missing 2x Specular Lobe Control

What UE4 has currently
activision-example-ue4

What is needed
activision-example

(Image from http://www.iryoku.com/downloads/Next-Generation-Character-Rendering-v6.pptx copyright of Activision)

It was hard to know how to setup the eye shader and hair as no documentation or example files existed. If anyone has any ideas on how to implement the Eye shader network in a more efficient usable manner it would be great to know, we can share what we know currently.

Eye Shader comparison and Layers

Notice reflection errors (white eyelash shadow) no caustics
eye-layers02

Correct reflections but no iris depth diffuse and no caustics
eye-layers03

(our eye shader uses a custom subsurface profile, with correct reflections but lacks caustics and correct deep inverted iris diffuse shading)

Some of our observations whilst using the eye shader
eye-errors

UE4 is truly incredible and a joy to work with. If this post is of any use we would be happy to share the models, project and custom skin shader with Epic or the market place so others can learn from them.

For eyes and hair we cannot get all the features to work without documentation but we got most things running. Any input from other users would be amazing. Being able to activate caustics and solve the incorrect eye occlusion (white eyelash reflection bug)  would be very beneficial. We would like to share what we have so far.

Without the extra correct specular controls it’s very hard to match the skin correctly when compared directly to real-world .exr reference data.

Shader Node Setup

skin_shader_instance skin_shader

Eye Node Shader

eye_shader_instance eye_shader

Eyelashes Setup

eyelashes_shader_instance eyelashes_shader

UE4 no doubt have more direct access into the engine’s core shader network, so we can only do so much as artists with the built in skin but it is already incredibly powerful.

Thank you Epic for such great tools, hopefully we can share with you our results so far, like we did with the Infinite head scan 🙂

25 Comments
  • ChenAReply

    ue4.11 add a character rendering example in the content examples in the market.
    or you can check the game paragon, epic developer the new skin and eye shader for this game.

    June 23, 2016
    • InfiniteReply

      This was based on that engine version but written before they released their eye pack.

      June 23, 2016
  • ChenAReply

    or you can put your sample in the market, i think many programmer will be interested in.
    because it's hard to get high quality model and texture.

    June 23, 2016
    • InfiniteReply

      Thanks, it's something I will think about.

      June 23, 2016
  • XodrocReply

    I would love to study your work, post on the UE4 forums too if you haven't! Much more likely to see improvements from Epic or programmers in the community.

    Also, wouldn't it be nice if you could use UDIM? Several 2K or 4K Normal maps instead of a single 8k or 16k.
    I haven't used it, but this plugin apparently lets you use UDIM in UE4 as well as maps up to 256K in size.
    http://graphinesoftware.com/products/granite-for-unreal

    July 3, 2016
    • InfiniteReply

      Hi, thanks. I had planned to post on the UE4 forums but didn't want to get blasted for posting such a heavy advertisement thread there. Or getting heavily critiqued as it's all WIP. I hope to revisit UE4 and skin again soon once IR's new scanner system is up and running. I was in contact with Epic about possible skin improvements but nothing further came from it. UDIM could be interesting for sure. Thanks for the link!

      July 6, 2016
  • HyunSeongSonReply

    Hi, I'm interested in IR skin shader! i sent mail, please reply me :)

    July 6, 2016
    • InfiniteReply

      Thanks for your interest. Sadly we can't share the shader as you would require a full custom engine version of UE 4.11 which is about 50GB. It might be best to start a thread on the UE4 forums or contact Epic about your request. It would be great if Epic could integrate SSS deep scatter into their Engine directly. Brian Karis is the main lead developer on digital skin there I believe.

      July 6, 2016
      • HyunSeongSonReply

        So sad, i wonder IR support Skin Shader services. I need help :)

        July 6, 2016
  • James AnswerReply

    Are you sure about the blue specular? I can see from your next-generation photometric scanning page that the difference between your polarized and non-polarized shots is indeed blue, but that doesn't really match with any literature I've read about skin rendering. The USC Digital Emily (http://gl.ict.usc.edu/Research/DigitalEmily/) project uses polarization too, but the shots are much more similar in hue and are missing the shift you have, so the extracted specular is largely monotone. Great work though, and I'd echo that I'd love to get hold of the assets, as having reference shots of easily matchable lighting is fantastic for verifying your shading.

    July 18, 2016
    • InfiniteReply

      Thanks for the post. If you check there Emily 2 data, you will see it has the exact same blue specular hue. From talking with others I believe it comes from the lamps used, in combination with the filters. If you remove the blue hue from the specular separated image and apply back with linear dodge, the skin looks dead. The blue is present and required. I'm not sure why others have not addressed it but it does seem important.

      July 18, 2016
      • James AnswerReply

        Hmm, I had a look at Emily 2 and I'm not seeing any blue in the exr, or the specular reference data.

        July 19, 2016
        • InfiniteReply

          They get exactly the same results they just choose not to show it by desaturating the results http://imgur.com/D83JhGE The blue hue is present. It depends on the colour calibration used. If you were to desaturate those results and add back, the skin will look flat and dead without that extra colour component.

          July 19, 2016
          • James Answer

            Interesting. I'm still not totally convinced it's not an artifact from the capture, but it wouldn't be the first time that observing that something looks better with some tweaks exposes problems with existing lighting models.

            July 20, 2016
  • ozelReply

    put ıt on marketplace ıll throw my money at ıt ı am very ınterested ın thıs

    July 18, 2016
  • FlyReply

    I'm very interested by your SSS shader, that awesome !!!

    July 20, 2016
  • LukasReply

    Is there any feedback from Epic yet? I would love to see backscatter in the editor!

    September 12, 2016
    • InfiniteReply

      Nothing. Best to talk with them. I hope they will include back scatter, it's integral.

      September 17, 2016
  • Ray LeReply

    Oh man, I love that SSS backscattering! I really want to know how did you do that in UE4. Is there any chance to learn this?

    September 12, 2016
    • InfiniteReply

      You would need a custom change made to the core engine. Not something we can help with sorry.

      September 17, 2016
      • jedliReply

        Hi,
        Really amazing work.
        Would you share how to implement the back lit transmit?
        I just guess that can calc the thickness "d" at ShadowProjectionPixelShader and save exp(-d*d) in one channel of shaodw buffer, and sample it at DeferredLightPixelShader. But under this you can only use 1 exp and can only do single transmit, from your result I notice multiple transmit was take in consideration. So maybe you can give some advice.
        For the skin and eye it looks much better and seems just need to change the material itself. Is it possible that share it?
        Thanks.

        February 20, 2017
        • InfiniteReply

          Hi you would be best to contact the author of the shader to ask https://twitter.com/xerxes1138

          February 20, 2017
          • jedli

            Thanks a lot.:)

            February 22, 2017
  • dvd.khReply

    is a chance to share this changed source files and this shader setup on github?

    November 22, 2016
  • Wesclei BarbosaReply

    Hi Lee,

    firstly congrats for the awesome results you are getting, really pushing the limits ....

    i am curious about your light match process... are you capturing a Hdri for each light conditions? i f so, how do you get the shadows???
    i have being using the xuan prada (http://www.elephantvfx.com/) method, for the past month and its give me a good result, using the chrome /gray balls and the color chart.
    could you share some infos about your method?
    thanks a lot

    December 12, 2016
Would you like to share your thoughts?

Leave a Reply

*