Ever since playing the Commandos series i've been fascinated with the way they displayed the view of the enemies. They had this triangular shape which would get cut off when geometry was blocking the view. I thought this was a great way of showing how much, and where, an enemy can actually see. You can see what im talking about in the below picture (the red shape). 

 

 

I still don't know exactly how they created the effect back then. Well, obviously I have a few ideas, but i'll never know exactly how. Anyway, I wanted this effect for a game I was writing, and so I had to figure out a way to create it. Incidentally it turns out this effect could work great for shadows/light as well.

First Steps

So, getting down to business. First off it is necessary to create a rendertarget which will be used to draw the visibility triangle and every mask onto. This allows us to resolve the texture from the rendertarget and thereby having separated this part from the actual scene.

In order to achieve the wanted effect we will have to draw the visibility triangle prior to drawing the masks, as they will, quite literally, be used to cut off pieces of the visibility triangle.

 

 

Looking at the above image, you can see how every pixel that is white is considered to be part of the "masked area". So, we can now draw the mask geometry (shadows if you will) for every wall in the scene. By drawing the mask geometry in the defined mask-color, we can achieve the effect we're going for. See the following diagram.

 

 

The above image shows how a mask is created, and drawn, for every wall in the scene (colored black only for illustrational purposes).

Constructing The Mask

All the information we need to know is the Start and End points (P0 and P1 respectively) of the line segment (wall) and the position of the dude that the visibility triangle is attached to.

With these pieces of information we already know the positions for 2 vertices of the mask, and can calculate the remaining 2 by finding the direction towards P0 and P1 from our dude's position. With these directions handy, we simply move along them and stop at a fitting time. Doing this, we should now know all 4 vertices of our mask, and are now able to draw it ontop of the visibility triangle.

Then, once we have the texture containing the visibility triangle and the full mask, we can draw a fullscreen quad and apply the texture to it using the following simple pixelshader (note that this is not a complete .fx shader. It is only showing the important parts):

 

float4 MaskColor = {1, 1, 1, 1}; // white

Texture VisibilityMask;

sampler maskSampler = sampler_state
{
    Texture = <VisibilityMask>;

        ...
};

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    float4 color = tex2D(maskSampler, input.TexCoords);
 
    float4 finalColor = color;
   
    if (color == MaskColor) {
        // make this pixel completely transparent
        finalColor.a = 0;
    }
   
    return finalColor;   
} 

 

Finally

The pixelshader function makes sure that every masked pixel gets drawn completely transparent, thus giving the "cut-off" effect.

 

On the above image you can see how a scene would look with masks drawn (left), and how it should look after cutting out the full mask (right).

So, to sum up; every Draw call the following should happen - in this order:

1. Clear designated rendertarget to the mask color
2. Draw visibility triangle
3. Draw masks (in same color as the specified mask color)
4. Get texture from rendertarget and save it
5. ---
   
6. Clear standard rendertarget as usual
7. Draw scene
8. Draw fullscreen quad and apply the saved texture (using a pixelshader that sets every masked pixel's alpha value to 0 - making it completely transparent)