Skip to content

Half precision (FP16)

Pass dtype="float16" to compute the formula in half precision. Inputs must be torch.float16 tensors.

import torch
import kernox as kp

device = "cuda"

x, y, b = kp.Vi("x", 3), kp.Vj("y", 3), kp.Vj("b", 1)
gauss = (kp.exp(-x.sqdist(y)) * b).sum_over("j")

run = kp.compile(gauss, dtype="float16")
X = torch.randn(1_000, 3, device=device, dtype=torch.float16)
Y = torch.randn(2_000, 3, device=device, dtype=torch.float16)
B = torch.randn(2_000, 1, device=device, dtype=torch.float16)
out = run(x=X, y=Y, b=B)                          # (1000, 1) float16

What is computed in FP16

  • The formula: every operation is rounded to FP16. Mathematical functions (exp, sin, ...) are computed in float32 and rounded to FP16, like PyTorch on the CPU.
  • Sums and the log-sum-exp family accumulate in float32 by default, then the result is rounded to FP16. Values beyond 65504 become ±inf.
  • Min, max and K-nearest reductions return FP16 values; indices and counts are int64.
  • Constants and Python parameters are rounded once to FP16. A constant above 65504 raises ConstantConversionError.

Reproducing PyKeOps exactly

PyKeOps accumulates FP16 sums in FP16, with block summation. To get the same behaviour, ask for an FP16 accumulator; sum_scheme then defaults to "block":

run_h = kp.compile(gauss, dtype="float16", acc_dtype="float16")
out_h = run_h(x=X, y=Y, b=B)

The float32 accumulator is the default in kernox because it is much more accurate for long sums, at a small cost.

Hardware

FP16 is supported on all three GPU backends. On NVIDIA cards older than compute capability 5.3, kernox emulates FP16 arithmetic (same results, slower). fast_math=True is not available in FP16.