Published : September 30, 2023
Talked your ear off about line rendering last time uh? Maybe you need some lighter reading? Fret not, for lines is not the only subject in expressive rendering, we also have shading!
In and of itself, shading is a big subject. It also has a lot of overlap with PBR, as it tends to be grounded in reality. But, as with all stylized rendering subjects, the difficulty is in how to use it to achieve a quality look.
In this article, instead of going into a deep multiple article dive, I'm going to take a more focused look at one of the very basic methods you may have already seen: toon shading!
While it is a basic building block, there's still a lot to learn, and a lot to expand on! I'll cover both the basics, limits of its use, and going beyond what's usually done with it!
The base idea of the method of toon shading is to have better control over how an object will appear in light or shadow. There are several ways to implement it, so I'll focus on the most straightforward one: toon ramps.
Toon ramps are one dimensional textures that match a value of light to a color. In order to do that, you'll need a parameter to represent that "light value", which is what I call the lighting coefficient. We'll see later on how to handle it, but for now we'll stick with the n.l. :If you don't know what that is or wish for a refresher, click here.
This is a bit generic, so let's illustrate a bit by building some ramps. Let's start with a neutral one and a constant one to ensure our algorithm works properly. We can make them directly in GIMP or another image editor, as 256x1 images.
Let's think for a bit. A neutral ramp would need to have increasing values for each pixel, so the 2nd pixel's value is (2,2,2), the third would be (3,3,3), and so on. It's a simple linear gradient. On the other hand, the constant needs the same value everywhere. So let's do just that and see the results:
Let's expand a bit more, and build a ramp to simulate two-tone shading. In order to build it, we need to decide where the threshold is, so to simplify I put it in the exact middle.
Let's put some colour into it! I'll use a light yellow for the lit part, and darker blue for the shaded part. It's as simple as putting those colours in the ramp.
Let's do another one to finish, by adding a gradient to each part, and by making the boundary a bit smoother by blurring it.
Hopefully this helped you understand that a bit better how it works! We could continue for a long while, it would be like trying to illustrate how colours can be used. Some other forms of toon ramps can be used, for instance by combining it with an albedo to not have to make a new one per object. This is also valid, and we'll see those a bit later! Some can also be :made using math (click to expand)
The n.l (pronounced "n-dot-l") is a major component of the rendering equation. It's a mathematical term representing how much the surface faces the light. Here's a quick breakdown:
One aspect to keep in mind, is that while the n.l is always between -1 and 1, most rendering applications will cut the -1 to 0 part since they are by definition unlit. In stylized rendering, we might want to keep that negative part, but it might affect our light accumulation step, as we'll see later in the article.
This part is a bit more theoretical, but you clicked on the link with 'math' in the title so you kinda had it coming.
A more precise definition of a toon ramp would be an injective function over the real range [0, 1]. Therefore, you could use any mathematical function instead of a texture. You can either map each color to a specific function, or the global luminance and then multiply it by an albedo (as we'll see later).
There are a lot of mathematical functions you may use here, so you can probably find the funky polynomial or Bézier curve of your dreams. Instinctively, I would look towards a sigmoid function, as it has a nice curve that allows us to go from shaded to lit quickly, while also having a nice ease of control with its inflection point being at 0.
The potential advantages of this method is potential speed if your function is quick enough, and additional resolution since you can do it at any scale. A disadvantage can be the difficulty of control, again depending on your function and interface. Of course, you may always bake the function into a texture to go back to the regular toon ramp performance.
Sometimes you see "toon shading" and "cel-shading" used interchangeably, but cel-shading is a specific subpart of toon shading, as it refers to using a limited palette to shade, often with only two or three tones. This is derived from hand animation, where in order to avoid differences from frame to frame, they would only use one color for a full zone.
Of course, in a lot of modern styles, you may see a smooth transition between the shaded and lit area, or an additional line, making it not strictly cel-shading in that sense. However, it is conceptually the similar and faces most of the same problems. The edges cases where it matters are a bit beyond the scope of this article.
Two handpainted examples of cel-shading. The left image uses strict cel-shading, while the one on the right includes some smooth transitions (like around Escha's eye). From Atelier: Escha & Logy
Thankfully, these are really simple to make, as it's literally only two colours around a threshold. Strict cel-shading may benefit from using
texelFetch instead of a regular texture call to ensure no smoothing takes places, while smooth cel-shading can be emulated by blurring the ramp.
In fact, those are simple enough that you can actually do it without a ramp, with a threshold and two colours as parameters. This is what I recommend if you are going to make actual cel-shading, as it will make your workflow and control easier.
The simplicity of two-tone shading however, brings a lot of attention to some issues that don't really appear with smooth PBR shading, and that is the control of the position and shape of the shadows. Indeed, since PBR shading doesn't change that much from one pixel to the other, we don't notice the slight differences or sharp discontinuities due to triangle interpolation, but the threshold we use will bring them out full force.
Left: Unappealing shading made obvious due to the threshold operation. Right: Adjusted shading giving a cleaner look. Figure credit to Colin Barton, who has worked plenty on the subject!
This problem is still a big one, long to explain, as well as still being the subject of academic research, so I'll probably talk about shadow control and cel-shading in depth another time, but this is beyond this shorter article.
This control over the appearance is nice, but it does hide a simplification that we need to understand, and that is hidden inside the lighting coefficient. Up to now in this article we have used the n.l to good effect, and it can indeed be sufficient, but it has a few issues.
One issue is that you have a limited shading area, as in PBR we cut off everything under 0. One way to fix this is to remap the n.l from the [-1, 1] range to the [0, 1] range with this simple formula:
lightingCoefficient = 0.5*nDotL + 0.5. This is known as the half lambert, and has been used in various games including TF2, but combined with toon ramps, this allows you to control the shading of the unlit part.
Another issue of n.l as a lighting coefficient is that it does limit material appearance, with for instance not being able to do rim lighting or specular highlights, since they are dependent on the view vector. This may be fixed by using the result of a shading model, such as the Blinn-Phong model, and converting that result to a lighting coefficient. More complex models may be used, but if you're going for a simple artstyle maybe you probably don't need fancy microfacet models.
However, using a shading model will also give you color data, which highlights another issue of simple toon ramps: how do you manage colour and coloured light ? The longer answer is complex and out of the scope of this introduction, but the simple answer is to use the luminance formula, which gives a single value for "how bright is that colour?" based on its RGB values. It is equal to
0.2126*R + 0.7152*G + 0.0722*B, but you may use other ones too. This will do the work pretty well in practice.
There is however still a big issue we have circled around, and that is managing multiple lights. In PBR, this is easy: you just add them. That's how reality works. But here, we are manipulating that reality, so just like most video games it will break when you give it weird values. We need to rethink our approach.
The part where it breaks is the light accumulation step, where you take the results of each light and add them. If we keep it the same, it will just take our already transformed colours and add them, making them really bright. Whoops! The solution is thankfully not hard, but the order of operations is really important, as it can change the result.
The first part to think about is how do you combine lights? As we said, in PBR it's a simple addition, but toon ramps I would recommend looking at the max operator instead. This will ensure the light remains in more expected outcomes, by basically just "changing the key light" as you go near it.
Another question, more specific to cel-shading, is when do you threshold? This one doesn't have a clear answer, it depends on your style. Thresholding before the accumulation will keep your highlights separate, while doing it after will combine them. Which one looks better will depend on your exact scene, but I think thresholding before has better results because you don't get smaller secondary highlights coming from the overlap of the two lights, which tend to look a bit chaotic in simplified artstyles.
To finish on the subject, I'll list some of the primitives you have to link together:
You always need the shading model at the start, and either add or max to accumulate several lights. Here are a few examples I can give you to get started:
Of course, you can ignore of all that and only use one key light. This works really well for characters! This is the method a fair amount of games use, including Zelda Wind Waker and Guilty Gear Xrd. You'll however need another model for environments as they often need more lights, but it's usually okay to use non-toon ramp methods for it.
So now that you know some basics of simple toon ramps, let's look at how we can get some extra performance out of this! This is actually one of the first things I did in my computer graphics path, so they are pretty accessible.
Up to now we've done a toon ramp with only one axis: the lighting coefficient. We've seen some of the limits and solutions, but we can also get some new effects by using additional parameters. This technique has been explored in a 2006 paper called X-Toon by Barla et al. It's closer to tech art than pure rendering, which serves us well here.
One example would be distance. By mapping it to the second axis, you can achieve new effects like smoothing out the shading at a distance, or reducing it akin to LODs. This can help simplify the style a lot!
Another one explored there is mapping the incidence angle, which you probably saw around more as the fresnel or rim light if you come from a 3D artist background. This can allow you to remake said rimlight, or achieve similar effects.
A last one, this time not from the paper but my first shader, would be mapping the specular. I've used it since I was working with metallic surfaces (because it was for mechs, of course). I don't have screenshots of the base version anymore, but it allowed for more dynamic renders, as we'll see in the next section.
I however wouldn't do it like this nowadays, as I didn't understand lighting coefficients yet, so I mapped diffuse and specular from the Blinn-Phong model directly. Since specular and diffuse aren't independant, there's lost space on the toon ramp, and if I had to do it again I would reserve some space for specular highlights on the lighting coefficient (if doing cel-shading).
This does however highlight some limitations of bidirectional toon ramps: they take up more space on the disk, and depending on your parameters not all of it is used. If a standard toon ramp size is 256x1, a standard 2-axis toon ramp is 256x256, while a 3-axis one is 256x256x256. Each step increases the size exponentially, so you have to be careful with that. 256x256 however is still very manageable, so it's not as drastic, but 3-axis can be pretty big so you're better of doing something more reusable, like...
So this has also probably been done before (hopefully), but I haven't seen records of anyone doing it. Haven't looked too close either, but it's probably basic enough that it didn't get a specific mention. Since I don't know an "official name", I'll call it the Shading Ramp, it's good enough for this.
The difference between a classic toon ramp and a shading ramp, is that the former outputs a colour, while the second outputs arbitrary parameters. Of course, most of the time these will be about the colour.
Since you can't really list all uses, I'll give a few examples of what I've used it for so that you may build your own.
I won't be going over exact implementations of each
because I'm lazy because you'll understand better if you try and apply it yourself (yeah sounds right), but I can give you a few tips for different engines. Some engine specific information may be a bit inaccurate / inefficient because either I don't really use it (Unreal, Unity) or because I've gone in too deep on custom rendering pipelines and thus haven't needed to figure out the "standard" way (Godot, Blender). Still, I believe this will help you start out!
If you actually use any of these and want to complete this, don't hesitate to shoot me a mail!
And here you go, you now have a few basics down for toon ramps! I wanted to write this because, while still overall a simple subject, there are some hidden tricks to it, and they can be a bit long to explain. Hopefully this helps you in your own games and renders!
If you can understand all that, I believe you have a nice base to this part of tech art. Indeed, you can't really be comprehensive in guides like these, since it would be like describing what you can do with a brush, it's just too wide. Through smart use of these and other techniques, you can achieve a lot of artstyles, but you need that base intuition in order to innovate and find your way.
Of course this article is only touching the surface, but there are a lot more parts to this, including "how do you handle coloured light?", "how do you handle projected shadows?", and "how can you control the shading?". Some of these can be really complex, and there hasn't been a lot of artist-accessible literature on this as far as I'm aware. Still, I hope this will help you get started on this long path!
(Did you know I wrote the article in a few hours, then procrastinated for like a month or two for the images and the code needed to make them? Completely unrelated to this, I might start writing about something other than computer graphics sometime.)
I'm mostly focusing on lines since it's my PhD subject, but if you want me to explain a specific subject, or have feedback, don't hesitate to shoot me a mail! Until then, see you next time!
Enjoyed the article? Register for the newsletter to not miss the next one!