Cel Shading

The main Lumberyard page shows a screenshot from Think Big. How is cel shading accomplished in Lumberyard and what is the best method for achieving it?

@Nazerith , great attention to detail and observation on your part
for that Think Big comment! There a few different ways to achieve translucent
material types. I would recommend using Human skin shader that’s provided in
the Lumberyard material system. The key parameters to make a successful
semi-transparent objects are “SSS Index” and “Translucency
Multiplier” . Like other 3D tasks, lighting setup also plays an important
role in all tests and gives it a bit bloom (in time of day setting). Also,
do not forget to place Environment probe or turn on Global Illumination in
environment setting.

Please let me know if this helps and please share your
progress with us.

In the current version of lumberyard you can actually run ‘cel shading’ (or ‘toon shading’ as it is also referred to) by applying a command line settings.

The command line should be ‘r_ApplyToonShading = 1’. Turning it off will be by setting the value to 0.

Hope this helps

thanks for the detailed answer! I’m studing about the toon shading thing, just confused aobut the default ‘cel shading’ parameters, like: can i change the width/color of the outline?

from the picture below we can see the outline is not perfect, and seems no Anti-aliasing,how could I improve those features?

Besides, can I make a custom toon shader whick contains multiple passes to render the outline?

Hi Tony,

Currently the code is provided as a starting point for brave people like yourself.
The shader itself works in the Z-deferred pass based on the stored GBuffer.
Since the shader is based on lookup TH for shading (one part of it) and on correct edge detection based on depth and normal deviation, going forward you can add more parameters to control the sampling distance and amount of samples and as you suggested, you can choose to add additional passes for blurring, sharpening and adding width before finalizing the stage.

Good luck,


It’s all happens on TiledShading pass TiledShading.cfi line 1126

    float3          toonDiffAcc = calculateToonColor(diffuseAcc);
    float3          toonSpecAcc = calculateToonColor(specularAcc);
    float3          silouetteLineColor = float3(0, 0, 0);   // EXPOSE to user
    float           silhouetteCoeff = CalculateOutlineAndFeatureSilhouettes( DepthRT, NormalRT, linearZ, pixelCoord, viewVec, surfNormal );

    float3 finalCol = lerp( silouetteLineColor, toonDiffAcc * surfAlbedo + toonSpecAcc, silhouetteCoeff);
    float3 finalCol = diffuseAcc * surfAlbedo + specularAcc;

there is used sum of two different methods for this effect - normal silhouette + depth silhouette, you can take a look on shaderLib.cfi file on method CalculateFeatureSilhouettes and CalculateOutlineAndFeatureSilhouettes also there is a lot parameters what not exposed yet to user’s settings. So you can try to adjust they manually :slight_smile:

1 Like

This is really helped a lot! I’ll check this and keep learning, thanks again for sharing~!

Hi Fluffy, I just modified some propertie descriptions of shader illum.ext, but it didn’t make any different in LY engine material, the console showed: (AssetCatalog) - Registering asset “shaders/shadertest.ext” via AssetSystem message, but type is not set. didn’t know why like this, do I need do something else like compiling? but I didn’t change illum.cfx

somewhere on youtube was tutorial about CryFX from Themodman101, so you can take a look on this

and one thing what I want to mentioned: I think TiledShading pass executes not per materal basis, but as common pass for whole frame with using RT’s what was filled with some data after gbuff passes.

1 Like

I made and tested a example of exposing one param for common pass such as TiledShading
for this you’ll need made few changes in engine code

for example you want change outline color on some key button, actually in futher you can use new method to expose it into Lua and ScriptCanvas

	else if (channelId == AzFramework::InputDeviceKeyboard::Key::AlphanumericU)
		if (state == AzFramework::InputChannel::State::Began)
			gEnv->p3DEngine->SetToonOutlineColor(Vec3(1.0f, 1.0f, 1.0f));
		else if (state == AzFramework::InputChannel::State::Ended)
			gEnv->p3DEngine->SetToonOutlineColor(Vec3(0.0f, 0.0f, 0.0f));

Step 1. Add method - gEnv->p3DEngine->SetToonOutlineColor()
open I3DEngine.h and add on line 2182 two methods

	virtual Vec3 GetToonOutlineColor() const = 0;
	virtual void SetToonOutlineColor(const Vec3 toonOutlineColor) = 0;

Step 2.
open 3dEngine.h
add implementation into CClass
line 873

// We just starting exposing %ApplyToon params here (so in futher we can change exposed params in Components via cpp or via ScriptCanvas ebus, Lua)
	virtual Vec3 GetToonOutlineColor() const { return m_toonOutlineColor; };
	virtual void SetToonOutlineColor(const Vec3 toonOutlineColor) { m_toonOutlineColor = toonOutlineColor; };
	// end here

add variable
line 1108 Vec3 m_toonOutlineColor;

Step 3.
open 3dEngine.cpp line 435 need init value then sub system start up, otherwice %applytoon will fill all with black color.

	m_toonOutlineColor = Vec3(0, 0, 0);

Step 4.
open D3DTiledShading.cpp on line 1069 we need send our new param(g_OutlineColor) into compute shader

		// Add outline color
		Vec3 outlinecolor = gEnv->p3DEngine->GetToonOutlineColor();
		static CCryNameR paramOutlineColor("g_OutlineColor");
		Vec4 vParamOutlineColor(outlinecolor.x, outlinecolor.y, outlinecolor.z, 1);
		CShaderMan::s_shDeferredShading->FXSetCSFloat(paramOutlineColor, &vParamOutlineColor, 1);
		// End add outline color

Step 5. open shader file TiledShading.cfi
on line 386

float4 g_OutlineColor;

fix line 1130

float3          silouetteLineColor = g_OutlineColor.rgb;   // EXPOSE to user

now save code and run cmd

 lmbr_waf configure
 lmbr_waf build_win_x64_vs2015_profile -p game_and_engine

then try to use gEnv->p3DEngine->SetToonOutlineColor(Vec3(1.0f, 1.0f, 1.0f)); in some temporary component to test, you may use key button event for this test.
ofc you need to enable r_ApplyToon… = 1 param befor.

1 Like

this tutorial is really great, helped me better understand the LY/Cry shaders, thx!

Hi Fluffy,

OH many thanks for the testing! the gif is pretty! i’ll try this, appreciate for ur kind and patient.

But now I still trapped in compiling shaders, as the video last post linked, I’ve set up the config and everything in the editor, but don’t know how exactly use the compiler to compile a shader.

Do I need compile manually? saw the Doc said edtor would compile automatically, but my editor still nothing happened. It would show the changes of ext, but I need to restart the editor everytime after I modified sth in shaders.

For my .cfx, nothing happened, so sad…:joy:
Could you please help me with the compiling thing ~?

I do not use remote compiler, for all my tests I just use default shadercompiller settings with Editor.

when you adding new shader you need make a pair
for example:\dev\Engine\Shaders\CustomIllum2.ext (copy of Illum.ext)\dev\Engine\Shaders\HWScripts\CryFX\CustomIllum2.cfx (copy of  Illum.cfx)

after this you need add new file - CustomIllum2 to xml shader list\dev\Editor\Materials\ShaderList.xml

on next step - restart editor and made new mat with using new shader, open CustomIllum2.cfx with helps notepad++, here you can start write code and save, each new shader’s code changing immediately updates in Editor’s viewport, so you can see result in realtime.

here is a additional info about shaders. you can look on this : https://docs.aws.amazon.com/en_us/lumberyard/latest/userguide/mat-shaders-custom-dev-intro.html

1 Like

Hi Fluffy, I do really want to use the default shadercompiller. But this is the exact problem I met befroe, just followed your step again, and the material will not refresh in realtime, except I restart the editor. That’s why I wanted try remote compiling.

The version I using is 1.20, and I’m downloading the 1.21 to try again.

Well, recently I was tried to use the RemoteShaderCompiler with same setting as you are described/published on pictures, and it’s seems it’s working fine for me. Editor creates the list of requests of list invariants for few various uber-shaders and RemoteCompiler compile them all.

Do not know, but maybe your issue do not related directly with LY ?

1 Like

I think so. Now the main problem I met seems is I can not compile shaders in realtime. I tried LY 1.21, the same problem stopped me.

For using the defalut shadercompiller,

  1. I can’t change the .ext file in realtime. For example, I modify some discription of the properties, there would not be a update in the Material Editor shader’s discription, even I reopen the Material Editor. And the console would show the message:
    Unless I restart the LY, the changes would show.
  2. I can’t compile the .cfx in realtime, neither by default shadercompiller nor remote compiller. For example, I modify some property discriptions in .cfx, then save it in sublime/notepad++, it seems the compiller worked, but got something wrong, the console showed:

    But I don’t exactly know what does this mean.

I think this is normal behavior, take look on this video , before add some property they closing editor

hmm… i think something wrong with shader code.
try to copy just original illum shader and do not modify it at all, just make new name for it, then use it for new mat. If it will works then your problem - code.
All CryFX shaders is very heavy ubershaders with logic that use references on other files, so if you doing into wrong way, steps left or step right - you are got compiling errors ))

1 Like

Yeah I’ve checked this video~ I tried again, it could work just need to restart LY Editor.
What I thought u said shader would refresh is like U3D or UE4, when shader/material changed, just save in the material editor then the viewport would update without restart the Editor.exe. For now it’s fine casuse I haven’t modified shaders a lot~ Really hope LY could update the changes without restart the Editor.exe soon!

Actually files with *.ext extension it’s not the shader it’s just collection of flags that will be push or not then your shader will be compiled (depends on selected options).
real code for shaders stored in files what placed in dir HWScripts\CryFX, editor react only on editing of this files.

1 Like