Rendering with Render Layers

A standard render

This is a quick (and dirty) manual explaining the use of Render Layers. With Render Layers you can choose to render specific objects with specific material attributes. For example, to render only shadows cast by objects, specularity of shiny objects, depth of field of a scene, etc.

Ambient Occlusion

An ambient occlusion render

Rendering shadows in the corners.

• Select the objects you want to render only with ambient occlusion.
• Create in the Channel Box a Render Layer with your selected objects.
• Right-click in this layer and select "Attributes".
• Click on "Presets" and choose "Occlusion".
• Maya assigns to the selected objects a Surface Shader with a mib_amb_occlusion node connected.

You do not need any (indirect) lights for this render. If you have used Physical Sun and Sky you can do two things:
Easy way (not recommended)

• Just delete the Physical Sun and Sky

Non-destructive way (recommended)

• Open the Camera Atribute Editor of your render camera and unfold the "mental ray" tab.
  • Right-click on Environment Shader and choose "Create Layer Override"
  • Right-click again on "Environment Shader" and choose "Break Connection"
  • Repeat the same steps for the Lens Shader.
• Make a render. For adjusting and tweaking read Mental Ray Ambient Occlusion tutorial.

Diffuse

A diffuse render

Rendering only visible color of objects. A direct light source is needed to be able to see colors.

• Select the objects you want to render only with diffuse.
• Create in the Channel Box a Render Layer with your selected objects.
• Right-click in this layer and select "Attributes".
• Click on "Presets" and choose "Diffuse".

You do need a light source to see the object's colors.

• Select the light source in the scene.
• Open the Attribute Editor and uncheck "Emit Specular". Repeat for other light sources if exists.

If you have a Physical Sun and Sky you can do two things:
Easy way

• Delete it and place a directional light with the same rotation as the sunShape.

Hard way

• Open the Camera Attribute Editor of your render camera and unfold the "mental ray" tab.
  • Right-click on Environment Shader and choose "Create Layer Override"
  • Right-click again on "Environment Shader" and choose "Break Connection"
  • Repeat the same steps for "Lens Shader".
• Click on the sunShape (the directional light being the sun's direction) and open the Attribute Editor
  • Right-click on "Emit Specular" and choose "Create Layer Override".
  • Clear the checkbox "Emit Specular".
  • Open the "Custom Shaders" section in the "mental ray" tab.
  • Right-click on "Suppress all Maya Shaders" and choose "Create Layer Override".
  • Clear the checkbox "Suppress all Maya Shaders".
  • Right-click on "Light Shader" and choose "Create Layer Override".
  • Right-click again on "Light Shader" and choose "Break Connection".
  • Repeat the same steps for "Photon Emitter".
• Make a render.

Specularity

A specular render

Rendering highlights of objects. In general, highlights can be seen on shiny and rounded objects. Highlights can also be seen on flat surfaces when a bright light shines on it. More information about specular reflectivity.

• Select the objects you want to render only with specular or glossy reflectivity. Select also the light source(s) which will generate the highlights.
• Create in the Channel Box a Render Layer with your selected objects.
• Right-click in this layer and select "Attributes".
• Click on "Presets" and choose "Specular".

You do need a light source to make the objects shine.

• Select the light source in the scene.
• Open the Attribute Editor and uncheck "Emit Diffuse". Repeat for other light sources if exists.

If you have a Physical Sun and Sky you can do two things:
Easy way

• Delete it and place a directional light with the same rotation as the sunShape.

Hard way

• Open the Camera Attribute Editor of your render camera and unfold the "mental ray" tab.
  • Right-click on Environment Shader and choose "Create Layer Override"
  • Right-click again on "Environment Shader" and choose "Break Connection"
  • Repeat the same steps for "Lens Shader".
• Click on the sunShape (the directional light being the sun's direction) and open the Attribute Editor
  • Right-click on "Emit Diffuse" and choose "Create Layer Override".
  • Clear the checkbox "Emit Diffuse".
  • Open the "Custom Shaders" section in the "mental ray" tab.
  • Right-click on "Suppress all Maya Shaders" and choose "Create Layer Override".
  • Clear the checkbox "Suppress all Maya Shaders".
  • Right-click on "Light Shader" and choose "Create Layer Override".
  • Right-click again on "Light Shader" and choose "Break Connection".
  • Repeat the same steps for "Photon Emitter".
• Make a render. You will get a black image with white highlights or spots.

Depth of Field (Z-level)

A Z-level render

• Select the objects you want to render only with depth of field information.
• Create in the Channel Box a Render Layer with your selected objects.
• Right-click in this layer and select "Attributes".
• Click on "Presets" and choose "Luminance Depth":
  
• Maya assigns to the selected objects a Surface Shader with additional nodes connected.
• The Attibute editor should open with the Surface Shader attributed. It's recommended to rename this Surface Shader to something clear and useful, such as 'luminanceDepthSS':
  
• Op the Hypershade, find and select the Surface Shader and show the shader network (Graph network or 'Show input and output connections' )
• The network should look like this:
  
• Select the setRange node in the Work Area. Open the attributes (double click, or Ctrl + a)
• Maya determines the maximum distance by using information from the camera. In most cases this maximum distance is far too big, making the Z-depth map useless.
• RMB-click the first column value of Old Max and choose break connection:
  
• Enter a manual value to be used as max distance. This is in scene units and should be the furthest distance you want the Z-depth map to cover.

Use batch render to make the render if rendering in de Render View does not produce the Z-depth map. For unknown reasons it sometimes doesn't work with the normal render command.

Shadows

A render showing cast shadows

The output of this render is only visible in the alpha channel. Therefore reflections and visibility of unwanted objects should not be visible in the alpha channel. Only shadows should be visible in the alpha channel.

• Select the objects which should cast and receive shadows.
• Create in the Channel Box a Render Layer with your selected objects.
• Right-click in this layer and select "Attributes".
• Click on "Presets" and choose "Shadow".
• Select the objects which should receive shadows.
  • Assign a "Use Background" shader.
  • Open Window > General Editors > Attribute Spread Sheet. In the Render tab turn off "Visible In Reflections".
• Select the other objects which should cast shadows and should be invisible.
  • Assign a "Surface Shader".
  • Open Window > General Editors > Attribute Spread Sheet. In the Render tab turn off "Primary Visibility" and "Visible In Reflections".
• Change the render setting in "mental ray".
• Make a render. Use the alpha channel.

Different method

A render showing all cast shadows

The output of this render will look like an ambient occlusion render, but instead it will contain the shadow data. You don’t need the image in the alpha channel; the render is visible in the default RGB channel. This method can be a solution to the complex way of the previous method, where objects that cast shadows and receive shadows are being separated. Most objects will be casting shadows as well as receiving shadows. This method will include all shadows in a scene in one render.

• Create a new render layer with all objects and the light source(s).
• Open the Camera Attribute Editor of your render camera and unfold the "Environment" tab.
  • Change the background color to white.
• Select all transparent objects.
  • Open Window > General Editors > Attribute Spread Sheet. In the Render tab turn off "Primary Visibility".
• Select all non-transparent objects.
  • Assign a “Use Background”.
  • Open the Attribute Editor of the “Use Background”.
  • Set the Reflectivity to 0.

If you have a Physical Sun and Sky. Delete it and place a directional light with the same rotation as the sunShape. Make sure that you turn on the shadows.

• Make a render.

Masks

A render with a custom mask

Specific masks can be created in Maya and be used later in image editing software, such as Photoshop, to edit certain parts of an image specifically.

• Create a new render layer with all objects. Light sources are not necessary.

• Open the Camera Attribute Editor of your render camera and unfold the "Environment" tab.
  • Change the background color to black.
• Create two "Surface shaders".
  • Keep one Surface shader black.
  • Make the other Surface shader white.
• Assign the white surface shader to objects that are going to be edited later on.
  • Assign the black surface shader to all other objects.
• Use "Maya software" for the rendering of masks.
  • Make sure the the "Edge anti-aliasing" in the render settings is set to 'high quality' or 'highest quality'.

• Make a render