Outlining Post Process

I've been working on the (temporary simplified) screen where you select a ship.  Currently when you click a ray is cast into the scene's physics space and the first entity hit is pulled back.  That's the one you clicked on, basically.

To visually implement this I had simply tinted the selected entity red when it was drawn (from the model rendering shader):

 if (isSelected)
  output.Color.r = output.Color.r * 2;

Clearly that was a bit... simplistic, though it was enough at the time and it worked.  But now I needed something a little better.



I thought about the typical selection rings or whatnot that you see in many games.  But what I really wanted was a nice outline effect.  Maybe this could be used in the game, red for enemies, green for friendlies, etc.

Anyway, I knew there were a few ways to do this.  The first was simply drawing the model twice, once scaled up a bit and in a flat color, the second time normally on top of that.  The other is to draw the model with some sort of edge-detection algorithm to generate an outline, which you then draw over the model.

I decided to go with the latter as there are other nice things you can do with edge detection.  I went looking for examples and quickly found one in the nice post-processing tutorials at digitalerr0r, the site of Petri Wilhelmsen (there is quite a lot of very informative and helpful stuff to be found there; I'd actually read through them all previously, so I was familiar with them).  I did note that at Adventure in Code John Marquiss had ported some of the examples up to XNA 4.0, particularly the Cel-Shading example that I was interested in.

I was really only interested in the edge detection, so I skipped the cel-shading stuff.  The shader consists of only a pixel shader which takes an image and, err, detects edges.  Well, really what it does is detect sharp color transitions, which isn't quite the same.  Anyway, I simply iterate through all the active entities and draw any that are selected to a rendertarget.  Then I just do:

 Device.SetRenderTarget(OutputRT);

 spriteBatch.Begin(SpriteSortMode.Deferred, BlendState.NonPremultiplied, 
                   null, null, null, Effects["Outline"]);   
 
 spriteBatch.Draw(outlineRT, 
                  new Rectangle(0, 0, OutputRT.Width, OutputRT.Height), 
                  Color.White);

 spriteBatch.End();

The outlineRT is quite small compared to the backbuffer size (I tried various sized and 512x256 gives a nice look.  Anything smaller gets really blocky when scaled up).

Hm, not quite right.

But, this didn't quite work.  The real problem of course being that it was not detecting depth changes, but rather color changes.  I considered changing my code to write depth information out (I've already got all that for the deferred rendering, after all).

Instead I simply added another form of RenderBehavior, basically a subtype of RenderBehavior.Forward, this being RenderBehavior.ForwardFlat.  With a quick addition to my Forward rendering shader, the very simple ForwardFlat now renders a model in one flat color (all white, for what it is worth).

This is pretty near the earlier solution B, but like I said I wanted to keep the edge detection.  So now that I have what amounts to a solid cutout of the model, I run the edge on that.  Bingo, it worked perfectly!

Exactly what I wanted.

I could just scale and draw the flat render, it would probably have about the same effect.  I may switch to that.  I'll likely experiment; having the capability to do either post-process technique will probably be handy.