Finally, we consider how the model-view matrix affects normals. Normals are of interest only in eye coordinates, so the rules governing their transformation to other coordinate systems are not examined.
Normals sent to the GL may or may not have unit length. If normalization is enabled, then normals specified with the Normal3 command are normalized after transformation. Normalization is controlled with
and
void Disable ( enum target ) ;
with target equal to NORMALIZE. This requires one bit of state. The initial state is for normals not to be normalized.
A normal at a point defines a plane at that point. If the normal is and the point is , then for the point to satisfy the plane equation we must have
whence
or q = 0 if w = 0. Therefore, if the model-view matrix is M, then the transformed plane equation is
and the transformed normal is
If normalization is disabled, then the square root in equation 2.1 is replaced with 1. Otherwise, the square root remains as written. If is the upper leftmost 3x3 matrix taken from M, implementations may choose to transform to eye coordinates using
followed by equation 2.1.
Because we specify neither the floating-point format nor the means for matrix inversion, we cannot specify behavior in the case of a poorly-conditioned (nearly singular) model-view matrix M. In case of an exactly singular matrix, the transformed normal is undefined. If the GL implementation determines that the model-view matrix is uninvertible, then the entries in the inverted matrix are arbitrary. In any case, neither normal transformation nor use of the transformed normal may lead to GL interruption or termination.