<p>\documentclass{article}
\usepackage[utf8]{inputenc}</p>
<p>\title{Tech Report: Layer-based Algorithms for Real-Time Expressive Rendering in Video Games}
\author{Panagiotis Tsiapkolis}
\date{July 2022}</p>
<p>% Aquarelle, autre styles</p>
<p>% Concentré sur l&#39;aspect layers et comment ça ajoute des imperfections</p>
<p>% Tricks for rendering, control pipeline</p>
<p>% Refreshing</p>
<p>% Limits, constraints</p>
<p>\begin{document}</p>
<p>\maketitle</p>
<p>% TODO Rendre idées principales plus claires: Pipeline complète DAC et engine, art itératif et imperfections en ressortant, extension de ces idées aux imperfections générales, animation des imperfections</p>
<p>\section{Abstract}</p>
<p>% TODO Présentation workflow, et structure de pipeline qui permet des algos qui incorporent l&#39;aspect &quot;itératif&quot; de l&#39;art.</p>
<p>\section{Introduction}</p>
<p>Bringing new artstyles to video games is an arduous task, requiring both technical and artistic expertise. Some of the more sophisticated algorithms can even require advanced 3D graphics programming skills, which makes them inaccessible to most artists and productions, while integrating them in an engine and pipeline requires practical knowledge of productions.</p>
<p>Since I am in a position where I&#39;ve at least dabbled in each of those parts, I figured I could try my hand at taking an algorithm and adapting it for practical use. I am thus writing this document to share and explain my findings, which may hopefully help other people interested in integrating unusual artstyles in their games.</p>
<p>% TODO Ref montesdeoca, motomura 2014, physical light
This tech report covers part of the work I did on the side between 2020 and 2022, adapting and adding to the algorithm described by Montesdeoca et al in 2017, with the constraints of production using a game engine. The adaptations and changes were driven both by what I&#39;ve seen of artistic pipelines, and by observations made based on feedback by artists and literature on the subject.</p>
<p>% TODO Parler des projets KT et Aeianthos</p>
<p>% TODO Ref Kronian Titans
I&#39;ve been working on two main algorithms which use a lot of the same theoretical basis. The first one, is the watercolor style, which showcases well a lot of the concepts, but requires some work. The second style is the one I&#39;ve developed for Kronian Titans, which is based on various inspirations as well as my own drawing style. I&#39;ll cover the subtleties of both, the concepts they rely on, and how the asset pipelines works. Finally, I will describe additional avenues I think are worthwhile continuations, some that I might even be able to pursue during my PhD.</p>
<p>Also covered in this report are practical ideas and methods that are helpful to speed up production and achieve effects by relying on the shaders and pipeline. These include both shortcuts that work well with the shaders, and how to use them to produce elements that are typically hand drawn like UI.</p>
<p>%\subsection{Technical Context}</p>
<p>%- Presentation des travaux
%- But d&#39;avoir plus de styles graphiques intéressants dans les jeux
%- Récup de la recherche et amélioration basé sur artistes
%- Dans un contexte de moteur de jeu, donc plus limité</p>
<p>%- Contexte Godot / Blender</p>
<p>%- on a plusieurs axes, les algos, les techniques, et les interfaces
%- les besoins des jeux ont amené des contraintes supplémentaires
%- shortcuts and UI</p>
<p>\section{Technical Constraints}
% TODO Remettre en contexte de jeu vidéo</p>
<p>% - Godot 3
% TODO Ref article
\subsection{Godot 3&#39;s rendering}
Some of the main constraints on the algorithm came from Godot 3&#39;s own renderer, which doesn&#39;t have a built-in way to control render passes. After some tests and trials, I was able to find a way to leverage its abilities with the following restrictions:
\begin{itemize}
    \item Only one geometry pass for each object, with only 3 color channels (24bits), or 6 if doing two renders. Only the first of those renders has access to light information in the whole pipeline.
    \item No access to the depth buffer.
    \item Pipeline only works in game, must use a secondary one for editor use. % TODO REF
\end{itemize}
In the end, I made a small library to make the creation of those node trees easier for future projects in Godot 3. With Godot 4&#39;s new renderer on the horizon, it will be worth to revisit this topic as we might be able to remove some of the restrictions. % TODO REF</p>
<p>This was one of the main reason I adopted the material ID workflow, as I couldn&#39;t fit all the information I would need otherwise. This also means that these restrictions are the main obstactle to better and easier control.</p>
<p>\subsection{Blender / Malt restrictions}
%- Pipeline / Blender
% two times the work, must implement tools in blender</p>
<p>%- General considerations</p>
<p>\section{Rendering Techniques}</p>
<p>% TODO Intro</p>
<p>\subsection{Pseudo G-Buffers and Material IDs}</p>
<p>%- Système
% TODO Ref
The main pipeline structure at work here is a modified setup similar to deferred rendering. A first pass is made rendering all the geometry and containing the info needed for the second pass. Due to game engine constraints, the lighting is also computed in this first stage using an half-shading method for more control, and due to space constraints on the available buffers, the only information stored is a material ID and various flags. For the similarity with deferred rendering, I&#39;ve been calling this first stage the \emph{PG-Buffer} (Pseudo G-Buffer).</p>
<p>% TODO Ref section asset pipeline
The second pass will then for each pixel gather the data stored, and use it to fetch the needed data from an external material atlas. The exact structure of this material atlas depends on the needs to the specific algoritm, but often takes the form of several toon ramps per material, stored in one to three textures. The way these atlases are created is detailed in a later section.</p>
<p>In practice, other passes may be added before the final one. I&#39;ve added some at the beginning for composition, and some to speed up outline generation. These are explained in the next section.</p>
<p>%- Pk adopter, restrictions, et utilités
% TODO ref à partie précédente ?
The adoption of this system was motivated by two main reasons: Godot 3&#39;s technical restrictions, and the fact that my usual drawing style doesn&#39;t benefit a lot from texture detail as opposed to geometric detail, meaning that a lot of the design&#39;s parts could be represented by the same material.</p>
<p>%- difficultés à l&#39;usage
It however brought a fair amount of restrictions with it:
\begin{itemize}
    \item The lack of textures and general parameters to pass really limits the amount of effects that can be achieved.
    \item Every material being managed by the second pass makes implementing several algorithms more expensive than it should be, both in computation time and code complexity.
    \item The storage of materials is an issue, both for the creation side which must keep clean records of material usage, and the technical side which must implement
\end{itemize}</p>
<p>%- setup via UV (détaillié plus tard)
% REF herb cape ?
Some of these restrictions can be lessened depending on the final product&#39;s design. For instance, as mentioned earlier, my drawing style being in tune with these restrictions lessens their impact, and can instead take advantage of some particularities of the algorithm to lessen the burden at the 3D modelling stage. Another example is Kronian Titans being a fighting game, means that \emph{palette swapping} (changing the colors of a character) works naturally with the need to manage the storage of materials manually, meaning the development time is lessened. However, these methods only help in certain cases. The restrictions introduced by this system become even more apparent on bigger teams, where records must be kept to date and shared between artists.</p>
<p>\subsection{Layer-based rendering}
% rendu itératif, imperfections accumulées
% gestion des bords</p>
<p>% TODO Plein d&#39;images</p>
<p>% REF Montesdeoca, livre light and shadow (?)
One of the biggest improvements made by the algorithm compared to previous efforts is the use of a layer based structure for the final render, inspired by the iterative process of artists which adjust colors in several passes. Watercolor&#39;s nature of transparent glazes on top of another makes this effect visible quickly.</p>
<p>The basic idea behind this is to allow each pixel (or in our case, material) to specify a color and additional values (pigmentation, viscosity...) for each layer separately, and merge them together at runtime using a color mixing algorithm of choice. In watercolor&#39;s case, this is done by using a variation [Montesdeoca et al]&#39;s algorithm once per layer, including the edge darkening effects, but not the geometry based distortions.</p>
<p>The result at this point could be precomputed, however the main ability this structure gives is the individual deformation of layers. By adding a noise-based offset to each layer&#39;s texel fetch, it is possible to create gaps and overlaps between layer by way of imprecisions, similar to how an artist would on a subsequent pass. This results in more natural looking imperfections, that can bring out more subtle effects and techniques of watercolor, like in figure TODO. During feedback with artists, the results were said to &quot;be really good for wet-on-dry watercolor paintings&quot;.</p>
<p>% TODO Figure mélange gris bleu+jaune</p>
<p>% perf
Performance-wise, the use of global layers puts an upper bound on the amount of potential overlaps, allowing the algorithm to run in realtime. Each layer added adds an additional run of the layer&#39;s algoritm, which will take at least two texel fetches for the PG-Buffer and the material atlas respectively. In many cases, such as watercolor, we also need neighboring information to compute lines or pigment density. These cases are sped up by precomputing the information in a previous pass and fetching from that buffer instead, making the complexity of gathering neighboring information go from $O(layers * neighboring samples)$ to $O(layers + neighboring samples)$ for a very small precision loss.</p>
<p>% TODO Verif mon pc portable
In practice, the watercolor algorithm with 8 layers could run at a stable 120 FPS (Godot&#39;s cap) on an AMD Vega card at a 2560x1440 resolution and 60 FPS on a 2016 MSI GP62 6QE-258FR Leopard Pro laptop with an NVIDIA GeForce GTX 950M and a 1366x768 resolution. The geometry pass being a very simple computation, the algorithm scales in cost mainly with resolution and the amount of layers.</p>
<p>\subsection{Animation}
% noise, inspiré par animation par celluloide</p>
<p>% REF pierre pour les contraintes en anim ?
One of the main restrictions of expressive rendering for film and video games is going from static to motion, as a lot of the more simple methods will result in losing the appearance of flatness, or introduce unwanted motion. This is even more true for mediums which aren&#39;t typically animated, such as watercolor or stained glass, really limiting what styles can be used. This section presents some early experiments in using methods inspired by celluloid animation.</p>
<p>Traditional animation methods rely on redrawing only the moving parts of the picture.  In traditional animation, this process is usually done with gouache on celluloid, taking advantage of gravity to erase the brush marks and the transparency of the celluloid to allow for compositing.</p>
<p>Watercolor is ill-suited for this, as while it is possible to rehumidify parts of the paper, this will result in smears and damage on the paper after repeated uses. Furthermore, the traditional animation process is made to erase the various potential sources of noise, whereas watercolor gets a major part of its aspect from that same noise due to its potentially thin glazes and pigment movement.</p>
<p>% TODO Je crois j&#39;ai pas implémenté, ptet avancer plus avant de finir le papier
The approach used by these experiments is to conceptually treat the different objects as celluloid layers, to be &quot;drawn&quot; once and to keep their stylization until a redraw is requested. Since doing this literally would require a lot of buffers and memory, this is instead done by doing the stylization relative to the object&#39;s origin. This would be akin to &quot;cutting&quot; the paper the watercolor was made on and moving it around the screen to composite, with a bit more flexibility.</p>
<p>In order to make this feel natural, conceptual &quot;redraws&quot; only happen at synchronized intervals, meaning an object will request a redraw which will only happen at the next global opportunity, preventing too frequent refreshes and ensuring that objects don&#39;t update on interlaced clocks, two situations that cause discomfort and reduce the final quality of the render. A good empirical value for this is around 20Hz.</p>
<p>An additional important result of these methods is that they reduce the negative effects of simple methods like screen space noise, meaning they can be added to existing algorithms to improve their results with very few changes.</p>
<p>This requires a lot of considerations and pipeline work, meaning the full technique wasn&#39;t implemented yet. However, several important steps were already made.</p>
<p>\subsubsection{Partial refresh of objects}
% Layer swaps ; global model data
% TODO</p>
<p>%To support
The implementation of these methods rely on the ability to refresh objects or part of them. As such, this requires some programming both on the shader side, and the general object logic side.</p>
<p>The shader side is managed by storing some of the stylization information in the buffers. One universal way to do it would be to record an offset for the object in screen space, that would be used to track its movement and thus keep the stylization stable. However, due to Godot&#39;s storage constraints, this is unsuitable.</p>
<p>The method used then, is a global counter that allows an object to store an offset of sorts, that can be used by the shader to associate other noises to the layers. In order to keep the performance at a reasonable level, instead of using brand new noise, we simply swap the noise associated to each layer around depending on that stored offset, allowing for that variation at no performance cost. As such, the offset was set to one of 4 states for a few reasons: having only two would become noticable quickly, four states can be stored neatly in only two bits, and since it matches with the amount of layers it doesn&#39;t add any processing amount.</p>
<p>On the object side, there is a need to both track the object and to synchronize its redraws. The latter is done through a global singleton and a messaging system, where all objects may receive a message at specified intervals, at which they may refresh. This ensures no object updates independently, and helps to keep the illusion of a lower framerate intact.</p>
<p>For the tracking, I am currently only tracking the object&#39;s origin and ignoring their animation, due to development time constraints. Tracking the movement of the animation would allow for a more robust system, as would having a manual rating of animations, but the current result seems robust enough.</p>
<p>These values are then used to compute a &quot;tension&quot; value, representing the need to redraw the object. The heuristic works because the more an object moves around, the more beneficial it would be to redraw the object for an animator. When the tension crosses a threshold, the object is conceptually &quot;redrawn&quot; at the next synchronized opportunity. This is done by choosing a new offset randomly, excluding the current one.</p>
<p>After some tests, it became apparent that refreshing can be jarring if done infrequently enough, and as such an additional time constraint has been added, forcing a refresh of all objects regularly. In my algorithms, I set it to an empirical value of 3Hz.</p>
<p>\subsubsection{Variable refresh rate based on movements}
% Noise refresh global</p>
<p>One aspect of limited animation is to not redraw the part until it moves enough to justify it. This means all movements are not equal relative to this constraint, as a simple translation doesn&#39;t require a full redraw, but a rotation however would. This is compounded by the effects of perspective which depend on where the camera is.</p>
<p>In order to simulate this behaviour, I keep track of a &quot;tension&quot; variable, which rises differently depending on the type of movement done.</p>
<p>% TODO Il me semble je fais rien pour les persos lol
Both the character&#39;s movements and the camera&#39;s movements must be taken into account for this to work well, meaning additional work must go into the pipeline. As such, I&#39;ve changed the camera&#39;s interface to be able to declare movements based on type, allowing the user to precise intent instead of guessing from the transforms, making the system a bit more robust to errors and allowing for more precise answers to specific actions like cuts.</p>
<p>When the camera&#39;s tension reaches a threshold, the whole noise is refreshed. While it would be possible to constrain the refresh to objects, this requires additional development constraints as well as tracking all objects, including the background, so this method has been chosen instead for the simplicity afforded and for the variety brought from refreshing the noise fully. Individual object&#39;s tensions are also reset at that moment.</p>
<p>Applying these concepts results in a render that feels much more natural and agreeable to the eye thanks to its stability. Please refer to the supplemental video to see the effect in motion.</p>
<p>% TODO Supplemental video</p>
<p>\section{Asset Pipeline}</p>
<ul>
<li>uv</li>
<li>blender malt</li>
<li>vieux outils</li>
<li>transfer shaders malt -&gt; godot</li>
</ul>
<p>\section{Style overview and development}
% parler de détails techniques de chaque pipeline ici</p>
<p>\subsection{Watercolor (Aeianthos)}
% melange couleurs, distorsion</p>
<p>\subsection{Kronian Titans}</p>
<p>\subsubsection{Fighting game specific constraints}</p>
<ul>
<li>readable</li>
<li>palette swap</li>
</ul>
<p>\subsubsection{Shading mechas}</p>
<ul>
<li>lighting</li>
<li>smears</li>
</ul>
<p>\subsubsection{Shading Pilots}</p>
<ul>
<li>special style constraints</li>
</ul>
<p>\subsubsection{User Interface}</p>
<p>\section{Limitations and Future Work}</p>
<ul>
<li>material ID not super good for flexibility</li>
<li>uses with watercolor</li>
</ul>
<ul>
<li>check si on parle bien de tout le pipe + moteur + ui/shortcuts</li>
</ul>
<p>% Fusion avec prochaine partie</p>
<p>\section{Related Work}
% ???</p>
<p>% Comparaison avec montesdeoca, controle différent (pas de gestion de quelles parties font des gaps</p>
<p>% Comparaison aquarelle 1997, pas même algo de couleurs</p>
<p>\section{Future Work}
% Watercolor better mixing
% noise tracking ?
% Lighting effects
% Godot 4, malt
% rendu itératif
% better noise
% Actual water in watercolor
% Controle perspective
% VFX Use</p>
<p>% OLD -------------------------</p>
<p>%\section{Overview of the algorithms}</p>
<p>%\section{Layer system}</p>
<p>%\subsection{Main loop}
% deferred rendering then layers</p>
<p>%\subsection{Adjacency and data storage}
% additional buffers separator and density + palettes + separators</p>
<p>%\subsection{Authoring tools}
% Malt pipeline</p>
<p>%\section{Variable rate partial refresh}</p>
<p>% TODO: Noise tracking</p>
<p>%\section{Specific usages}</p>
<p>%\subsection{Watercolor}
% density, color mixing and pigmentation
%</p>
<p>%\subsection{Pencil}
% anisotropic noise
% maybe just as extended ?</p>
<p>%\subsection{Kronian Titans}
% anisotropic noise</p>
<p>%\section{Results}</p>
<p>% + thanks ?</p>
<p>\end{document}</p>