Skip to content

Kernel MMD

Kernel norms, also known as maximum mean discrepancies (MMD), compare two measures through a kernel k:

Loss(α, β) = ½⟨α, k⋆α⟩ + ½⟨β, k⋆β⟩ − ⟨α, k⋆β⟩

They need a single pass over the N × M pairs (no iterations), which makes them the cheapest losses of remblais. Their weakness is the gradient: far from the target, a Gaussian kernel barely pulls the points.

Built-in kernels

import torch
from remblais import SamplesLoss

device = "cuda"

x = torch.randn(2_000, 3, device=device)
y = torch.randn(2_500, 3, device=device) + 0.5

for name in ("gaussian", "laplacian", "energy"):
    loss = SamplesLoss(name, blur=0.5)
    print(name, loss(x, y).item())
  • "gaussian": k(x, y) = exp(−‖x − y‖² / 2σ²), with σ = blur;
  • "laplacian": k(x, y) = exp(−‖x − y‖ / σ);
  • "energy": k(x, y) = −‖x − y‖, the energy distance. It has no scale, so blur is ignored.

Weights, batches and gradients work as for the Sinkhorn divergence:

a = torch.rand(2_000, device=device)
a = a / a.sum()
b = torch.full((2_500,), 1 / 2_500, device=device)
x.requires_grad_()
L = SamplesLoss("gaussian", blur=0.5)(a, x, b, y)
[g] = torch.autograd.grad(L, [x])

Choosing the blur

blur sets the scale at which the kernel compares the measures. Details smaller than the blur are smoothed out; structures much larger than the blur are seen only through their tails. A good starting point is the typical distance between the two clouds.

Custom kernels

A custom kernel goes through loss="hausdorff" with a kernel argument, as in GeomLoss. With the tensorized backend, the kernel receives tensors of shape (B, N, D) and (B, M, D) and returns the (B, N, M) kernel matrix:

def cauchy(x, y, blur=0.5, **kwargs):
    d2 = ((x.unsqueeze(-2) - y.unsqueeze(-3)) ** 2).sum(-1)
    return 1 / (1 + d2 / blur**2)

loss = SamplesLoss("hausdorff", kernel=cauchy, blur=0.5, backend="tensorized")
print(loss(x.detach(), y))

With the online backend, it receives kernox formulas instead (x indexed by i, y by j), and blur as a number:

def cauchy_formula(x, y, blur):
    d2 = ((x - y) ** 2).sum()
    return 1 / (1 + d2 / blur**2)

loss = SamplesLoss("hausdorff", kernel=cauchy_formula, blur=0.5, backend="online")
print(loss(x.detach(), y))

The two values agree up to rounding. The kernel is compiled into a single fused reduction for the given blur.

Large clouds

backend="online" never stores the N × M kernel matrix. In dimension 1 to 3, backend="multiscale" also skips the blocks of pairs farther apart than truncate times the blur (default 5). Pass truncate=None to keep every pair.