How to create a shader

I’ve been updating this now and then, could use some feedback on it. The Illum and Post effect shader sections could use some additional info for sure.

How to create a shader

Shaders are computer programs that specify how to render objects and effects. There are two types of shaders used: lighting shaders that interact with scene illumination (e.g. Illum, Terrain), and regular shaders that don’t calculate any lighting information but for used for post-processing effects (e.g. PostEffects, PostEffectsGame) . All lighting shaders have a common structure and make use of a unified shading interface.

Lumberyard uses an ubershader system with compile-time defines to handle the many different shader permutations that are required for combining numerous shader features. The shader format used that is very similar to High-Level Shader Language (HLSL), DirectX FX, and CgFX.

There are three common shader file types:

Extension Description Location
.cfi Common shader code that can be included in other files using the #include directive Engine/Shaders/HWScripts/CryFX
.cfx The unique shader program for each shader. Multiple shader techniques can also be grouped into a single .cfx file. Engine/Shaders/HWScripts/CryFX
.ext Optional shader extension files with the same name as the corresponding .cfx file which define properties exposed to the material editor. Engine/Shaders

The properties defined by .ext files are called Shader Property Flag or Shader Generation Flag . There are 3 types of them:

Extension Other Names Location
Static Flags Engine Flags, Static Mask, ST Mask Engine/Shaders/Static.ext
Runtime Flags Runtime Mask, RT Mask Engine/Shaders/RunTime.ext
Global Flags Material Flags, Global Mask, GL Mask Engine/Shaders/.ext, all others

Shaders are compiled for faster loading into the following types

Extension Description
.cfib “CFI Binary”. Pre-tokenized version of .cfi files. Created during engine initialization.
.fxb .fxb - A collection of shader reflection information generated during the shader packaging process if r_shadersExport=1 is set in shadercachegen.cfg (which is default). This only occurs during the shader packaging process and is then only read if r_shadersImport > 0 at runtime (default disabled in Debug/Profile builds and default enabled in Performance/Release).
.fxcb “FX Compiled Binaries”. These are custom “pak” files of compiled shaders for specific shaders and their techniques. Files will be named similar to illum@common_zpassvs.fxcb showing that is is Illum.cfx using the common_zpassvs shader and all possible permutations that have been discovered for common_zpassvs are appended to the end of this file as they are discovered

During shader development, the renderer always has to load and parse all shader source files (.cfx and .cfi) in order to generate the engine-level reflection information for a shader. This includes generating which slots we bind textures, samplers and constant buffers for a given shader permutation, as well as what techniques, passes, parameters and a variety of other metadata the renderer uses for a given shader permutation. This step is rather expensive and should be avoided for performance and release builds by using binary shaders only in shader paks. Binary shaders are created as part of the shader packaging process by the ShaderCacheGen tool. Internally, generation and usage are controlled with the following CVARs.

CVAR Description
r_shadersImport 0 = All .cfx and .cfi files are parsed at runtime to determine shader reflection information. 1 = Import pre-parsed shader reflection information from .fxb files if they exist for a related .cfx which skips expensive parsing of .cfx files in RT_ParseShader. If a .fxb exists for a shader but an individual permutation is missing, then fallback to the slow .cfx parsing for that permutation. 2 = .fxb files are required for all shaders and no fallback is allowed. We will never parse .cfx/.cfi files at runtime. Caveat - if r_shadersAllowCompilation=1 is set and r_shadersImport=2, then r_shadersAllowCompilation will win and turn off r_shadersImport because compiling new shaders may generate new .fxb files. 3 (default) = For release and profile builds use same behavior as 1. For all other build configurations use mode 0.
r_shadersExport 1 = export .fxb files during shader cache generation

Other useful shader files include:

File Description
dev/Engine/Shaders/Shaders.sln A Visual Studio solution with all the engine shaders.
dev/Editor/Materials/ShaderList.xml Contains a list of shaders used to populate the Material editor list of shaders.

Anatomy of a shader file

The first part of a shader .cfx file typically includes the 'Script" section.

// Shader global descriptions
float Script : STANDARDSGLOBAL
<
    string Script =
        // Determines if the shader will be made visible to UI elements
        // For example, if set the shader will be available in the Material Editor
        "Public;"
        // If set when SupportsFullDeferredShading is not set, we go through the tiled forward pass
        // In the tiled forward case, we will execute the pass defined in this file - and not Common*Pass
        "SupportsDeferredShading;"
        // Determines that the shader supports fully deferred shading, so will not go through the forward pass
        "SupportsFullDeferredShading;"
>;

After the script declaration, optional .cfi files are included with the #include directive. For example:

#include "ShadeLib.cfi"

Next, are optional sections defining custom structs, samplers and tweakables.

// Custom shading pass structure /////////////////
struct fragPassCustom
{
};
 
struct fragLightPassCustom
{
};
 
 
// Custom texture sampler /////////////////
#if %SOME_FEATURE_FLAG
    sampler2D MySampler
    {
        string UIName = "My Map";
        string UIDescription = "My Map description.";
        Texture = $Opacity;
    };
#endif // %SOME_FEATURE_FLAG
 
 
// Tweakables /////////////////
#if %SOME_PROPERTY
half SomeParameter
<
    register = PER_MATERIAL_12.x;
    string UIHelp = "Controls some aspect of the shader";
    string UIName = "Some Param";
    string UIWidget = "slider";
    float UIMin = 0.0;
    float UIMax = 10;
    float UIStep = 0.005;
> = 2.0
#endif // %SOME_PROPERTY

Custom tweakable properties like the %SOME_PROPERTY tweakable above are defined in an optional .ext file that has the same name as your custom shader. For example, the Illum.cfx shader has a corresponding Illum.ext extension file. These properties are exposed in the material editor.

An example custom property in an extension file looks like this:

Property
{
  Name = %SOME_PROPERTY
  Mask = 0x100
  Property    (My Property Name)
  Description (Creates a shader permutation that does fancy stuff!)
}

The tweakable section is followed by the definition for the shader implementations and technique.

vert2FragGeneral MyVertexShader(app2vertGeneral IN)
{
  vert2FragGeneral OUT = (vert2FragGeneral)0;
  streamPos vertPassPos = (streamPos)0;
  vs_shared_output(IN, OUT, vertPassPos, true);
      
  return OUT;
}
   
pixout MyPixelShader(vert2FragGeneral IN)
{
    pixout OUT = (pixout)0;
    OUT.Color.rgb = GetDiffuseTex(diffuseTex, IN.baseTC);
    OUT.Color.a = 1;
    return OUT;
}
 
 
technique General
<
  string Script =
        "TechniqueZ=ZPass;"
        "TechniqueZPrepass=ZPrepass;"
        "TechniqueMotionBlur=MotionBlurPass;"
        "TechniqueCustomRender=CustomRenderPass;"
        "TechniqueShadowGen=ShadowGen;"
        "TechniqueDebug=DebugPass;"
>
{
    pass p0
    {  
        VertexShader = MyVertexShader() GeneralVS;
        PixelShader = MyPixelShader() GeneralPS;
        ZEnable = true;  
        ZWriteEnable = true;  
        CullMode = Back;
    }
}

Example: To create a simple shader

  1. Create a new file named Example.cfx in dev/engine/shaders/hwscripts/cryfx or duplicate the ReferenceImage.cfx shader and rename the duplicate.
  2. Open the file in a text editor and enter the following code

Example.cfx

#include "Common.cfi"
#include "ShadeLib.cfi"
#include "vertexLib.cfi"  

float Script : STANDARDSGLOBAL
<
    string Script =
        "Public;"
        "ShaderType = General;"
        "ShaderDrawType = General;"
        "AfterHDRPostProcess;"
        "ForceDrawLast;"
>;
 
vert2FragGeneral ExampleLightVS( app2vertGeneral IN )
{
    vert2FragGeneral OUT = (vert2FragGeneral) 0;
 
    streamPos vertPassPos = (streamPos)0;
    vs_shared_output(IN, OUT, vertPassPos, false);
 
    return OUT;
}
 
float4 ExampleLightPS( in vert2FragGeneral IN ) : COLOR
{
    float4 cColor = GetDiffuseMap( diffuseMapSampler, IN.baseTC );
 
    cColor.rgb = pow(cColor.rgb, 1.0 / 2.2 );
    cColor.w *= GetInstance_Opacity();
 
    return cColor;
}
 
technique General
{
    pass p0
    {
        VertexShader = ExampleLightVS();
        PixelShader = ExampleLightPS();
    }
}

Use the r_reloadshaders command to reload all shaders or restart the Editor. This simple shader can now be used on an object by creating a new material in the Material Editor, selecting the Example shader from the shader drop down, providing a diffuse map and then assigning the material to an object in the scene.

Example: To create an Illum-based shader

The simple example above does not interact with lights, shadows or any post-processing. In order to make a shader that works with the general PBR pipeline:

  1. copy dev/engine/shaders/hwscripts/cryfx/Illum.cfx, dev/engine/shaders/hwscripts/cryfx/CommonZPass.cfi and dev/engine/shaders/hwscripts/cryfx/CommonZPrePass.cfi
  2. in your copy Illum.cfx shader copy rename IlluminationVS/PS/GS
  3. at the bottom modify the includes so they include your copy of CommonZPass.cfi and CommonZPrePass.cfi
  4. usually you have to make changes to your copy of CommonZPass.cfi to make changes to diffuse/ambient lighting, blending, etc and there are matching changes in your copy of Illum.cfx and some additional stuff like emissive lighting.

Example: To create a post-effects shader

  1. Create a new class that extends CPostEffect in dev\Code\CryEngine\RenderDll\Common\PostProcess\PostEffects.h
  2. Register the effect by adding the appropriate AddEffect(CMyEffect); in CPostEffectsMgr::Init in dev\Code\CryEngine\RenderDll\Common\PostProcess\PostProcess.cpp
  3. Put the implementation of your new effect in the appropriate PostProcess.cpp in dev\Code\CryEngine\RenderDll\XRenderD3D9 i.e. dev\Code\CryEngine\RenderDll\XRenderD3D9\PostProcessMisc.cpp or dev\Code\CryEngine\RenderDll\XRenderD3D9\PostProcessGame.cpp
    NOTE: if you create your own post effect shader group be sure to add it in include the new group .xml file in your seedlist file (should be in /libs/posteffectgroups/<your_file>.xml ). The default post effects group is in dev/Engine/Libs/PostEffectGroups/Default.xml
  4. Add your shader code to an existing .cfx file already loaded by the engine like dev\Engine\Shaders\HWScripts\CryFX\PostEffects.cfx or dev\Engine\Shaders\HWScripts\CryFX\PostEffectsGame.cfx or make your own .cfx file and be sure to create a CShader like the static ones in CShader.h and call gRenDev->m_cEF.mfRefreshSystemShader(“YourShaderFileName”, CShaderMan::s_shMyShaderName) which will actually load the shader.
  5. By default your shader will now render, but if you want to be able to turn it off/on, override the Preprocess() method and return false. For example, this is how the CFilterSharpening decides to render or not:

PreProcess code

bool CFilterSharpening::Preprocess()
{
    bool bQualityCheck = CPostEffectsMgr::CheckPostProcessQuality(eRQ_Medium, eSQ_Medium);
    if (!bQualityCheck)
    {
        return false;
    }
    if (!CRenderer::CV_r_PostProcessFilters)
    {
        return false;
    }
    if (fabs(m_pAmount->GetParam() - 1.0f) + CRenderer::CV_r_Sharpening + CRenderer::CV_r_ChromaticAberration > 0.09f)
    {
        return true;
    }
    return false;
}

Example: To create a compute shader

TODO - sorry haven’t written this one yet, but Lumberyard supports and relies on compute shaders for some of the heavy lifting in the render pipeline. Search for ‘ComputeShader’ in the dev/engine/shaders/HWScripts/CryFX folder to see examples (tiled shading, volumetric fog, SVOGI, post processing)

Shader editing workflow for general shaders

  1. restart the editor after creating the initial shader (or use r_reloadshaders)
  2. create a material that uses that shader and apply it to an object
  3. set ed_backgroundupdaterate 100 in the editor console so the editor will update in the background while you have your text editor in focus
  4. make changes to your shader in the text editor of your choice and save the shader file (don’t need to close it)
  5. the shader hot reloads in the editor every time you save so it’s pretty fast to iterate and debug

Notes

  • your shader .cfx and .cfi files can live in a gem if you put the files in dev/Gems//assets/shaders/hwscripts/cryfx you can also put your .ext files (if you have new ones) in dev/Gems//assets/shaders Keep in mind, if the name of your shader is the same as one of the shaders in dev/engine/shaders then your gem’s shader will overwrite the engine one when it is copied to the cache. This can be useful for all sorts of things, or a pain depending on what you’re trying to do.
  • if you duplicate the illum.cfx shader and modify the fragment shader your changes to the color output won’t take affect because by default color is determined in the zprepass gbuffer fragment shader which is not in illum.cfx
  • if you duplicate the ReferenceMaterialHDR.cfx shader you will note that increasing the emissive intensity on a material that uses the shader makes the object transparent. Also, changing the opacity/alpha value of the material causes incorrect depth sorting and no actual transparency. To resolve the later, add ZEnabled = true, ZWriteEnable = true and CullMode = Back to the pass.
  • setting ed_backgroundUpdatePeriod to a value greater than 0 (100 is fine) will make it so you don’t have to switch back and forth between the editor and your shader text editor.
16 Likes

@petrocket Good, honestly I say to you , It is not enough, If you agree or If you can, please making video tutorials for this new issue :wink: Thank you :rose: :pray:

2 Likes

Thanks for posting this! Maybe I can make time this weekend and record this on my free time.

4 Likes

This is Gold @petrocket thanks for posting this!.

1 Like

Something to add for custom post process shaders - if you create your own post effects group .xml file, you need to include it in your seedlist file (should be in /libs/posteffectgroups/<your_file>.xml)

1 Like

Thanks @SlothyGames, great addition!

Thanks for posting this Fantastic!!!