Autotuning and wisdom¶
For each reduction, a backend can generate several kernel variants (block
size, how the work over j is split, how the K-nearest lists are shared
between threads...). By default kernox picks one with a fixed heuristic. You
can instead let it measure a few variants on your GPU and remember the
fastest, in the spirit of FFTW's planning levels and "wisdom".
import torch
import kernox as kp
device = "cuda"
x, y = kp.Vi("x", 3), kp.Vj("y", 3)
knn = kp.compile(x.sqdist(y).argkmin(16, over="j"), dtype="float32", tuning="measure")
X = torch.randn(2_000, 3, device=device)
Y = torch.randn(5_000, 3, device=device)
idx = knn(x=X, y=Y) # first call of this size: measures, then runs
idx = knn(x=X, y=Y) # later calls: no measurement
print(knn.last_decision)
Tuning levels¶
tuning |
Behaviour |
|---|---|
"estimate" (default) |
heuristic choice, no measurement; results are reproducible on a given device |
"wisdom_only" |
use a stored decision if there is one, otherwise "estimate"; never measures |
"measure" |
use a stored decision, or measure up to 4 variants (about 2 s at most) |
"patient" |
same, with up to 32 variants (about 30 s at most) |
Changing variant never changes the meaning of the result: same dtypes, same tie and NaN rules. Only float sums can differ slightly, because the terms are added in a different order.
When measurements happen¶
- Only on the first call for a given size class. Sizes are grouped by factors of about √2 for M, N, the number of batch elements and the density of block-sparse ranges. Other calls in the same class reuse the decision without measuring or recompiling.
- The measuring call is blocking: it synchronizes with the GPU. It measures on the real inputs of the call (read-only), and the call returns the result computed with the chosen variant.
- If the timings are too noisy (for example because the GPU is busy), the decision is kept for this compiled object only and not stored.
Measure on an idle GPU, so that the stored decisions reflect the real speed.
Inspecting decisions¶
run.last_decision describes the decision of the last call: the level, its
source ("estimate", "wisdom", "measured" or "fallback"), the chosen
variant, the size class, whether it was stable and stored, and the measured
times. run.tuning_decisions lists all decisions of a compiled object.
A backend without variants (such as the CPU oracle) falls back to
"estimate" and logs a message once.
Wisdom files¶
Stored decisions ("wisdom") are JSON files, one per device, in
$KERNOX_CACHE_DIR/wisdom (by default ~/.cache/kernox/wisdom). A decision is
only reused on the same device (identified by its UUID when the driver
reports one), with the same driver, compiler and kernox code generator, and
only by a level at most as demanding as the one that produced it: "patient"
does not trust a decision made by "measure".
To save, restore or reset the wisdom:
data = kp.export_wisdom() # all valid entries, as a dict
kp.export_wisdom("wisdom.json") # the same, written to a file
n_imported = kp.import_wisdom("wisdom.json") # also accepts the dict
n_deleted = kp.forget_wisdom() # deletes the wisdom files
import_wisdom skips invalid entries and returns the number of entries
imported; a file of another wisdom format raises ValueError.