Cel-Shading in lwjgl Thread

hello everyone, im currently developing a 3d game using java and lwjgl and right now im just trying to get all the shaders and lighting done they way i want. i got the cel-shading done in my glsl shaders, but now im trying to get the black outlining done(im pretty sure everyone knows, but just in case, cel-shading is a kind of shading in video games that uses solid colours, minimal textures, and black outlines around objects to make the game look cartoon-is). There are a couple different methods for outlining 3d objects that i know of. the one my game is currently using is called fixed funcitonality. basically, if you wanted to add a treasure chest to your game, you would set glPolygonMode(GL_BACK, GL_LINE), and then cull the front faces. you set the line width to something pretty high, like 3.5f. then, you render the same model a second time, this time in glPolygonMode(GL_FRONT_AND_BACK, GL_FILL) and disable the culling. it works, but it doesnt look too pretty. the lines are very jaggedy looking. There is another method of getting the outlining done, its whats called an "Image space method". in this method you use a FrameBuffer and sobel edge detection to produce an outline of the scene. then you combine the outline image with the original FrameBuffer. this is the method that i want to use, but i cant find any good tutorials, and often the code that i find doesnt work in lwjgl, so i cant figure it out. anyways, if any of yall could teach me how to get this done, that would be super awesome. thanks,

-Roochie

Edited by: Roochie

A quick search online uncovered this GLSL fragment shader for sobel operation that looks sensible enough:

// sobel.fs
//
// Sobel edge detection

uniform sampler2D sampler0;
uniform vec2 tc_offset[9];

void main(void)
{
    vec4 sample[9];

    for (int i = 0; i < 9; i++)
    {
        sample[i]= texture2D(sampler0, 
                              gl_TexCoord[0].st + tc_offset[i]);
    }

//    -1 -2 -1       1 0 -1 
// H = 0  0  0   V = 2 0 -2
//     1  2  1       1 0 -1
//
// result = sqrt(H^2 + V^2)

    vec4 horizEdge = sample[2] + (2.0*sample[5]) + sample[8] -
                     (sample[0] + (2.0*sample[3]) + sample[6]);

    vec4 vertEdge = sample[0] + (2.0*sample[1]) + sample[2] -
                    (sample[6] + (2.0*sample[7]) + sample[8]);

    gl_FragColor.rgb = sqrt((horizEdge.rgb * horizEdge.rgb) + 
                            (vertEdge.rgb * vertEdge.rgb));
    gl_FragColor.a = 1.0;
}

Blitzbasic.com -> Starstonesoftware.com

Here is a link to the GLSL docs if you get stuck: Khronos.org

I've only ever used Sobel for sharpening. If the texture on the model had edges you might get an outline you dont want.

Wouldnt it be better to use a stencil buffer to get an outline of the model?

thanks for your comments guys, and shinobiNFC, i didnt know exactly what a stencil buffer was(im not super familiar with image manipulation code) so i searched it up. it appears that a stencil buffer actually would be loads better for getting the outline of a model. Thank you! now i just need to learn how to use stencil buffers haha. if yall know of any helpful tutorials or reads, please feel free to share

Hmm unfortunately my only experience reading about it was in the blue book (or possibly the red book). It seems to be documented on the lwjgl forums though.

Sorry I cant be of more help.

I've ordered the openGL superbible off amazon yesterday so ill let you know if theres anything about stencil buffers in there

superbible is blue book right? If so that's the good one. Red is more legacy

EDIT: You got the good one

Edited by: ShinobiNFC

its is blue but im not sure if its the blue book either way i've heard good things about it, also got Game Engine architecture which ive seen referenced a few times around the web and has positive reeviews

yeah game engine architecture is a great book. One of my favourites.

i dont really understand the Stencil Operations portion of the OpenGL Bible, so i dont think i can be of any help with that at the moment, id probably have to read through the begining chapters first

this is basically whats in the book Flipcode.com