Gradients¶
kernox reductions are differentiable with torch.autograd. If an input tensor
has requires_grad=True, the result is part of the autograd graph, and
backward() computes the gradients with fused reductions: no M×N matrix
appears in the backward pass either.
A first gradient¶
import torch
import kernox as kp
device = "cuda"
x, y, b = kp.Vi("x", 3), kp.Vj("y", 3), kp.Vj("b", 1)
s = kp.Param("s")
energy = kp.compile((kp.exp(-x.sqdist(y) * s) * b).sum_over("j"), dtype="float64")
X = torch.randn(1_000, 3, device=device, dtype=torch.float64, requires_grad=True)
Y = torch.randn(2_000, 3, device=device, dtype=torch.float64, requires_grad=True)
B = torch.randn(2_000, 1, device=device, dtype=torch.float64)
S = torch.tensor([0.5], device=device, dtype=torch.float64, requires_grad=True)
out = energy(x=X, y=Y, b=B, s=S)
out.sum().backward()
print(X.grad.shape, Y.grad.shape, S.grad.shape) # (1000, 3) (2000, 3) (1)
Gradients are available with respect to Vi and Vj tensors, and to Param
values passed as tensors. A Param passed as a Python number has no gradient.
How it works¶
kernox differentiates the formula symbolically with symox, then runs the
derivatives as new sum reductions on the same backend: one over j for the
gradients with respect to Vi inputs and parameters, one over i for the
Vj inputs. These gradient reductions are built on the first backward() and
then reused.
Gradients have the dtype of the inputs. With acc_dtype="float64", the
gradient sums also accumulate in float64.
What can be differentiated¶
| Reduction | Gradient |
|---|---|
sum_over |
first and second order |
logsumexp, logsumexp(weight=), sumsoftmaxweight |
first order |
min, max, kmin, and the values of min_argmin, max_argmax, kmin_argkmin |
first order, through the selected terms |
argmin, argmax, argkmin, counts |
none: integer outputs, as in PyTorch |
For min, max and kmin, the gradient flows through the selected pair only, as in PyKeOps; kernox evaluates the derivatives on those K pairs per row.
A second derivative of a reduction other than sum_over raises an error from
PyTorch.
Second order¶
For sums, the backward pass is itself differentiable. Use create_graph=True,
for example for a Hessian-vector product:
X2 = X.detach().requires_grad_(True)
(gx,) = torch.autograd.grad(energy(x=X2, y=Y, b=B, s=0.5).sum(), X2, create_graph=True)
v = torch.randn_like(X2)
(hv,) = torch.autograd.grad((gx * v).sum(), X2) # Hessian-vector product, (1000, 3)
torch.autograd.gradcheck and gradgradcheck work in float64.
Non-smooth points¶
| Formula | Derivative used |
|---|---|
where(c, a, b) |
derivative of the active branch; the condition is not differentiated |
abs(f) |
sign(f), so 0 at 0 |
minimum, maximum, clamp at a tie |
1/2 for each side |
floor, ceiling, sign, round, step |
0 |
sqrt(f) at 0 |
inf, as in PyTorch |
An invalid value in the inactive branch of a where does not contaminate the
gradient:
d2 = x.sqdist(y)
r = kp.where(d2 > 1.0, kp.sqrt(d2 - 1.0), 0.0)
run = kp.compile((r * b).sum_over("j"), dtype="float64")
X3 = X.detach().requires_grad_(True)
run(x=X3, y=Y.detach(), b=B).sum().backward()
assert torch.isfinite(X3.grad).all()
Gradients of a formula inside a formula¶
To use a derivative as part of a formula (a force field, a score function, a
Laplacian), build it with f.grad(v, e), f.laplacian(v) and friends: see
Formula operations.