Numerics¶
kernox aims for results you can reason about: by default it follows IEEE floating-point arithmetic, and anything that trades accuracy for speed is an explicit option.
What is guaranteed by default¶
- No fast-math:
exp,log, divisions and square roots are the correctly behaved library versions. - No reassociation of your formula, no flush of subnormal numbers to zero, no TF32, and no silent conversion of float64 to float32.
- Every constant is rounded exactly once, to the reduction's dtype.
- Only one freedom is left to the compiler: fusing
a * b + cinto a single fused multiply-add (FMA), described below.
The order in which the terms of a sum are added is not specified and depends on the GPU and on the kernel variant, so float sums can differ slightly from one backend or variant to another.
Constants¶
Constants are exact until the reduction is compiled:
- Python
intandfractions.Fractionvalues stay exact integers and rationals; afloatkeeps its exact binary64 value; symox numbers such aspistay symbolic. symox folds constant subexpressions exactly. - At compile time, each floating-point constant is rounded once, to nearest
(ties to even), to the dtype. Rationals are emitted as
p/qin the dtype. - A finite constant that overflows the dtype, or a non-zero constant that
would round to zero, raises
ConstantConversionError.infandnanwritten explicitly are kept. - Parameters passed as Python numbers follow the same rule. A
Parampassed as a tensor must already have the exact dtype.
Fused multiply-add¶
By default (fp_contract=True), the compiler may compute a * b + c inside
one expression as an FMA, with a single rounding instead of two. This is what
nvcc and PyKeOps do. It is not less accurate, but results can differ in the
last bit between backends.
For bit-for-bit reproducibility across backends, forbid it:
import torch
import kernox as kp
device = "cuda"
x, y = kp.Vi("x", 3), kp.Vj("y", 3)
d2 = x.sqdist(y)
run = kp.compile(kp.exp(-d2).sum_over("j"), dtype="float64", fp_contract=False)
Summation schemes¶
Sums (and the internal sums of the log-sum-exp family) accept three schemes,
the same as PyKeOps' sum_scheme:
sum_scheme |
How terms are added | Error, roughly |
|---|---|---|
"direct" (default) |
one running sum per row | grows like N · u · Σ|term| |
"block" |
sums of blocks of T terms, then the total |
(N/T + T) · u · Σ|term| |
"kahan" |
compensated summation (Kahan-Babuška, also known as Neumaier) | 2 · u · Σ|term|, independent of N |
Here u is the unit roundoff of the accumulator (about 6e-8 in float32).
"kahan" costs a second accumulator per output; it is the scheme to use when
a float32 sum over a million terms must stay accurate.
precise = kp.compile(kp.exp(-d2).sum_over("j"), dtype="float32", sum_scheme="kahan")
The other reductions (min, max, indices, counts) ignore sum_scheme.
A wider accumulator is the other way to improve a sum: with
dtype="float32", acc_dtype="float64" accumulates in float64.
Special values¶
| Reduction | Neutral value (no term) | NaN terms | Ties |
|---|---|---|---|
| sum | 0 | propagate | |
| count | 0 | a comparison with NaN is false | |
| min / max | +inf / -inf |
ignored | |
| argmin / argmax | -1 |
ignored | smallest index |
| kmin / argkmin | padding with +inf / -1 |
ignored | smallest index first |
| logsumexp | -inf |
propagate | |
| sumsoftmaxweight | NaN (0/0) | propagate |
- Infinities are ordinary values: a
+infterm can be returned bymintogether with its index. - When
Kis larger than the number of candidates,kminandargkminpad with+infand-1. - Log-sum-exp is computed in one pass with the usual shift by the running
maximum, so large scores (
1e4) do not overflow. A term with score-infhas zero weight. A single+infscore gives+infforlogsumexpand NaN forsumsoftmaxweight.
where and invalid values¶
where(cond, a, b) returns exactly the value of the active branch. An invalid
operation in the other branch, such as sqrt(-1), log(0) or 1/0, does not
leak into the result, and does not leak into gradients either.
safe = kp.where(d2 > 1.0, kp.log(d2 - 1.0), 0.0)
X, Y = torch.randn(100, 3, device=device), torch.randn(50, 3, device=device)
out = kp.compile(safe.sum_over("j"), dtype="float32")(x=X, y=Y)
assert torch.isfinite(out).all()
Simplifications made by symox¶
Formulas go through the automatic simplifications of symox, which assume exact
real arithmetic. For example x / x becomes 1, and ~(a < b) becomes
a >= b. This matters with NaN: ~(a < b) is then false when a is NaN,
whereas the literal negation would be true. Write the comparison you mean
rather than negating its opposite.
Approximate exp and log¶
compile(..., fast_math=True) allows approximate exp and log, the
counterpart of PyKeOps' use_fast_math. It is off by default and never turned
on implicitly. See fast_math.
Half precision¶
FP16 has its own rules (float32 accumulator by default, float16 output): see FP16.