OpenGL Picking Tutorial

Picking, or selecting, a particular item in a 3D scene may prove useful for some applications. The selection can be performed by clicking on an object, requiring a way to determine over which object was the mouse placed.

A simple solution to achieve this is to use color coding, i.e. to render the scene using a routine that draws each pickable object with a specific color. Reading the pixel where the mouse is located provides the color, and hence, the ability to identify the object.

The rendering in selection mode uses a very simple shader that applies a constant color to the pixels. The color is a uniform variable that should be set with a unique value before each object is drawn.

The vertex shader could as follows:

#version 330

uniform mat4 m_pvm;

in vec4 position;

void main()
{
	gl_Position = m_pvm * position ;
}

The fragment shader is as simple as:

#version 330

uniform int code;

out vec4 outputF;

void main()
{
	outputF = vec4(code/255.0, 0, 0, 0);
}

It is important that the code we receive is an integer and not a float. When in RGB mode there are only 256 possible values for each component, so when you set a color using floats OpenGL will pick the nearest color possible, which may not be exactly the value you provided.

Note that we divide by 255.0, not by 255. Using the latter approach would be an integer division and the result would also be an integer (either 0 or 1).

If more than 255 objects are available for selection then the uniform code could be a ivec4.

Suppose our scene has 4 chess pawns, as shown in the following figure:

In this example we have four clickable objects, which we are going to assign codes from 1 to 4. The background of the selection rendering routine can be set be black, hence zero means nothing is picked. When using the selection routine we would get an image as follows (highly enhanced contrast):

This image will never be presented to the user as the selection rendering does not swap the buffers.

The following routine receives the mouse window coordinates and executes the necessary steps to determine which object was picked. It starts by calling the selection rendering routine, more on this latter. Afterwards, it reads the pixel from the back buffer and checks the returned color.

To read the pixel we’ll use function glReadPixels.
[stextbox]void glReadPixels(GLint x, GLint y, GLsizei width, GLsizei height, GLenum format, GLenum type, GLvoid *data);

Parameters:

• x,y: the coordinates of the first pixel of the rectangular block to be read
• width, height: the size of the block.
• format: the format of the pixel data: GL_STENCIL_INDEX, GL_DEPTH_COMPONENT, GL_DEPTH_STENCIL, GL_RED, GL_GREEN, GL_BLUE, GL_RGB, GL_BGR, GL_RGBA, GL_BGRA
• type: the data type of the pixel components: GL_UNSIGNED_BYTE, GL_BYTE, GL_UNSIGNED_SHORT, GL_SHORT, GL_UNSIGNED_INT, GL_INT, GL_HALF_FLOAT, GL_FLOAT, GL_UNSIGNED_BYTE_3_3_2, GL_UNSIGNED_BYTE_2_3_3_REV, GL_UNSIGNED_SHORT_5_6_5, GL_UNSIGNED_SHORT_5_6_5_REV, GL_UNSIGNED_SHORT_4_4_4_4, GL_UNSIGNED_SHORT_4_4_4_4_REV, GL_UNSIGNED_SHORT_5_5_5_1, GL_UNSIGNED_SHORT_1_5_5_5_REV, GL_UNSIGNED_INT_8_8_8_8, GL_UNSIGNED_INT_8_8_8_8_REV, GL_UNSIGNED_INT_10_10_10_2, GL_UNSIGNED_INT_2_10_10_10_REV, GL_UNSIGNED_INT_24_8, GL_UNSIGNED_INT_10F_11F_11F_REV, GL_UNSIGNED_INT_5_9_9_9_REV, or GL_FLOAT_32_UNSIGNED_INT_24_8_REV
• data: the returned pixel data.

[/stextbox]

void processSelection(int xx, int yy) {

	unsigned char res[4];
	GLint viewport[4]; 

	renderSelection();

	glGetIntegerv(GL_VIEWPORT, viewport);
	glReadPixels(xx, viewport[3] - yy, 1,1,GL_RGBA, GL_UNSIGNED_BYTE, &res);
	switch(res[0]) {
		case 0: printf("Nothing Picked \n"); break;
		case 1: printf("Picked yellow\n"); break;
		case 2: printf("Picked red\n"); break;
		case 3: printf("Picked green\n"); break;
		case 4: printf("Picked blue\n"); break;
		default:printf("Res: %d\n", res[0]);
	}
}

To get the correct pixel we must convert the mouse window coordinates, origin on the top left corner, to frame buffer coordinates, origin on the bottom left corner. That requires that we know the height of the viewport. Function glGetIntegerv can be used for this purpose, with GL_VIEWPORT as the first parameter. The return variable, viewport, is an array with 4 items that provides the x and y viewport window coordinates (origin: top left) followed by width and height of the viewport. Then, the pixel can be read, and the retrieved color properly decoded.

The selection rendering routine must follow the same steps as the regular routine regarding geometric transformations, so that objects are in the same place on screen. In general, the selection rendering routine is a simplified version of the regular rendering routine, where only a subset of objects are drawn, and no graphical effects, such as lighting, are setup.

The subset of objects includes all clickable objects plus occluders where relevant. The idea is that if some of the chess pawns were inside a room, invisible to the user, then they should not appear in the color coded image. This can be easily achieved by drawing the occluders (for instance the walls of the room) with the same color as the background.

void renderSelection(void) {

	glClearColor(0.0f, 0.0f, 0.0f, 0.0f);
	glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

	//set matrices to identity
	...
	// set camera as in the regular rendering function
	....

	// use the selection shader
	glUseProgram(selectionProgramID);

	//perform the geometric transformations to place the first pawn
	...
	// set the uniform with the appropriate color code
	glProgramUniform1i(selectionProgramID, codeVarLocation, 1);
	// draw first pawn
	...
	
	// repeat the above steps for the remaining objects, using different codes

	//don't swap buffers
	//glutSwapBuffers();

	// restore clear color if needed
	glClearColor(1.0f, 1.0f, 1.0f, 1.0f);
}

Downloads:

• VS2010 project with full source code and shaders (ZIP)

The project requires glew, and freeglut or glut.