pickRay function

bool pickRay(
  1. Matrix4 cameraMatrix,
  2. num viewportX,
  3. num viewportWidth,
  4. num viewportY,
  5. num viewportHeight,
  6. num pickX,
  7. num pickY,
  8. Vector3 rayNear,
  9. Vector3 rayFar,
)

On success, rayNear and rayFar are the points where the screen space pickX, pickY intersect with the near and far planes respectively.

The viewport is specified by (viewportX, viewportWidth) and (viewportY, viewportHeight).

cameraMatrix includes both the projection and view transforms.

Returns false on error, for example, the mouse is not in the viewport.

Implementation

bool pickRay(
    Matrix4 cameraMatrix,
    num viewportX,
    num viewportWidth,
    num viewportY,
    num viewportHeight,
    num pickX,
    num pickY,
    Vector3 rayNear,
    Vector3 rayFar) {
  bool r;

  r = unproject(cameraMatrix, viewportX, viewportWidth, viewportY,
      viewportHeight, pickX, viewportHeight - pickY, 0.0, rayNear);
  if (!r) {
    return false;
  }

  return unproject(cameraMatrix, viewportX, viewportWidth, viewportY,
      viewportHeight, pickX, viewportHeight - pickY, 1.0, rayFar);
}