kaolin.vision¶

project_points(pts: torch.Tensor, intrinsics: torch.Tensor, extrinsics: torch.Tensor = None)[source]¶

Projects a set of 3D points onto a 2D image, given the camera parameters (intrinsics and extrinsics).

Parameters

pts (torch.Tensor) – 3D points to be projected onto the image (shape: \(\cdots \times N \times 3\) or \(\cdots \times N \times 4\)).
intrinsics (torch.Tensor) – camera intrinsic matrix/matrices (shape: \(B \times 4 \times 4\) or \(4 \times 4\)).
extrinsics (torch.Tensor) – camera extrinsic matrix/matrices (shape: \(B \times 4 \times 4\) or \(4 \times 4\)).

Note

If pts is not of dim 2, then it is treated as a minibatch. For each point set in the minibatch (of size \(B\)), one can apply the same pair of intrinsics or extrinsics (size \(4 \times 4\)), or choose a different intrinsic-extrinsic pair. In the latter case, the passed intrinsics/extrinsics must be of shape (\(B \times 4 \times 4\)).

Returns: pixel coordinates of the input 3D points.
Return type: (torch.Tensor)

Examples

>>> pts = torch.rand(5, 3)
tensor([[0.6411, 0.4996, 0.7689],
        [0.2288, 0.9391, 0.2062],
        [0.4991, 0.4673, 0.6192],
        [0.0397, 0.3477, 0.4895],
        [0.9219, 0.4121, 0.8046]])
>>> intrinsics = torch.FloatTensor([[720, 0, 120, 0],
                                    [0, 720, 90, 0],
                                    [0, 0, 1, 0],
                                    [0, 0, 0, 1]])
>>> img_pts = kal.vision.project_points(pts, intrinsics)
tensor([[ 720.3091,  557.8361],
        [ 919.0596, 3369.7185],
        [ 700.4019,  633.3636],
        [ 178.3637,  601.4078],
        [ 945.0151,  458.7479]])
>>> img_pts.shape
torch.Size([5, 2])

>>> # `project_points()` also takes in batched inputs
>>> pts = torch.rand(10, 5, 3)
>>> # Applies the same intrinsics to all samples in the batch
>>> img_pts = kal.vision.project_points(pts, intrinsics)
torch.Size([10, 5, 2])

>>> # Optionally, can use a per-sample intrinsic, for each
>>> # example in the minibatch.
>>> intrinsics_a = intrinsics.repeat(5, 1, 1)
>>> intrinsics_b = torch.eye(4).repeat(5, 1, 1)
>>> # Use `intrinsics_a` for the first 5 samples and
>>> # `intrinsics_b` for the last 5
>>> intrinsics = torch.cat((intrinsics_a, intrinsics_b), dim=0)
>>> img_pts = kal.vision.project_points(pts, intrinsics)
>>> img_pts.shape
torch.Size([10, 5, 2])

>>> # Can also use a per sample extrinsics matrix
>>> pts = torch.rand(10, 5, 3)
>>> extrinsics = torch.eye(4).repeat(10, 1, 1)
>>> img_pts = kal.vision.project_points(pts, intrinsics, extrinsics)

unproject_points(pts: torch.Tensor, depth: torch.Tensor, intrinsics: torch.Tensor)[source]¶

Unprojects (back-projects) a set of points from a 2D image to 3D camera coordinates, given depths and the intrinsics.

Parameters

pts (torch.Tensor) – 2D points to be ‘un’projected to 3D. (shape: \(\cdots \times N \times 2\) or \(\cdots \times 3\)).
depth (torch.Tensor) – Depth for each point in pts (shape: \(\cdots \times N \times 1\)).
intrinsics (torch.Tensor) – Camera intrinsics (shape: :math:` cdots times 4 times 4`).

Returns

Camera coordinates of the input points.: (shape: \(\cdots \times 3\))

Return type

(torch.Tensor)

Examples

>>> img_pts = torch.rand(5, 2)
tensor([[0.6591, 0.8643],
        [0.4913, 0.8048],
        [0.2129, 0.2338],
        [0.9604, 0.2347],
        [0.5779, 0.9745]])
>>> depths = torch.rand(5) + 1.
tensor([1.4135, 1.0138, 1.6001, 1.6868, 1.0867])
>>> intrinsics = torch.FloatTensor([[720, 0, 120, 0],
                                    [0, 720, 90, 0],
                                    [0, 0, 1, 0],
                                    [0, 0, 0, 1]])
>>> cam_pts = kal.vision.unproject_points(img_pts, depths, intrinsics)
tensor([[-0.2343, -0.1750,  1.4135],
        [-0.1683, -0.1256,  1.0138],
        [-0.2662, -0.1995,  1.6001],
        [-0.2789, -0.2103,  1.6868],
        [-0.1802, -0.1344,  1.0867]])
>>> cam_pts.shape
torch.Size([5, 3])

>>> # Also works for batched inputs
>>> img_pts = torch.rand(10, 5, 2)
>>> depths = torch.rand(10, 5) + 1.
>>> cam_pts = kal.vision.unproject_points(img_pts, depths, intrinsics)
>>> cam_pts.shape
torch.Size([10, 5, 3])

>>> # Just like for `project_points()`, can use a per-sample intrinsics
>>> # matrix.
>>> intrinsics_a = intrinsics.repeat(5, 1, 1)
>>> intrinsics_b = torch.eye(4).repeat(5, 1, 1)
>>> # Use `intrinsics_a` for the first 5 samples and
>>> # `intrinsics_b` for the last 5
>>> intrinsics = torch.cat((intrinsics_a, intrinsics_b), dim=0)
>>> cam_pts = kal.vision.project_points(img_pts, depths, intrinsics)
>>> cam_pts.shape
torch.Size([10, 5, 3])