Half precision (FP16)¶
Pass dtype="float16" to compute the formula in half precision. Inputs must be
torch.float16 tensors.
import torch
import kernox as kp
device = "cuda"
x, y, b = kp.Vi("x", 3), kp.Vj("y", 3), kp.Vj("b", 1)
gauss = (kp.exp(-x.sqdist(y)) * b).sum_over("j")
run = kp.compile(gauss, dtype="float16")
X = torch.randn(1_000, 3, device=device, dtype=torch.float16)
Y = torch.randn(2_000, 3, device=device, dtype=torch.float16)
B = torch.randn(2_000, 1, device=device, dtype=torch.float16)
out = run(x=X, y=Y, b=B) # (1000, 1) float16
What is computed in FP16¶
- The formula: every operation is rounded to FP16. Mathematical functions
(
exp,sin, ...) are computed in float32 and rounded to FP16, like PyTorch on the CPU. - Sums and the log-sum-exp family accumulate in float32 by default, then
the result is rounded to FP16. Values beyond 65504 become
±inf. - Min, max and K-nearest reductions return FP16 values; indices and counts are int64.
- Constants and Python parameters are rounded once to FP16. A constant
above 65504 raises
ConstantConversionError.
Reproducing PyKeOps exactly¶
PyKeOps accumulates FP16 sums in FP16, with block summation. To get the same
behaviour, ask for an FP16 accumulator; sum_scheme then defaults to
"block":
run_h = kp.compile(gauss, dtype="float16", acc_dtype="float16")
out_h = run_h(x=X, y=Y, b=B)
The float32 accumulator is the default in kernox because it is much more accurate for long sums, at a small cost.
Hardware¶
FP16 is supported on all three GPU backends. On NVIDIA cards older than
compute capability 5.3, kernox emulates FP16 arithmetic (same results, slower).
fast_math=True is not available in FP16.