Conformance suite¶
The conformance suite (tests/conformance/) checks every GPU backend against
the same FP64 reference, with the same rules. It is shared by the cuda,
hip and sycl backends. A backend without a PyTorch device or without its
module is skipped, never simulated.
Running¶
# fast profile (CI) on all available backends
python -m pytest -m "not slow" tests/conformance
# long profile
python -m pytest -m slow tests/conformance
# restrict the backends, force fuzzing seeds
KP_CONFORMANCE_BACKENDS=hip KP_FUZZ_SEEDS=0:500 python -m pytest tests/conformance/test_fuzz.py
These are correctness tests: they may run on a shared GPU, but never during a benchmark on the same card (they would skew its measurements).
Contents¶
| File | Role |
|---|---|
_support.py |
backend discovery, inputs, FP64 reference, error bounds, judgment |
test_matrix.py |
systematic matrix (reductions × dtypes × dimensions × K × sizes, over="i", acc_dtype) |
test_adversarial.py |
NaN, ±inf, −0.0, denormals, huge/tiny values, duplicates, constants, where guarding invalid domains |
_fuzz.py, test_fuzz.py |
seed-based formula generator, minimization, recorded regressions (regressions/*.json) |
test_keops_port.py |
targeted port of PyKeOps tests (provenance in THIRD_PARTY.md) |
test_where_cse.py |
region-based CSE: NaN, ±inf, 1/0 in the inactive branch, against the FP64 reference |
test_autograd.py |
gradients of SUMs: dense Torch (FP32/FP64), FP64 gradcheck and gradgradcheck |
test_execution.py |
execution rules: non-default stream/queue, zero elements, destruction during execution, no host copy, multi-GPU |
test_softmax.py, test_reductions_v08.py |
LOGSUMEXP family, MIN_ARGMIN, MAX_ARGMAX, special values |
test_reduction_gradients.py |
gradients of the LOGSUMEXP family and of MIN, MAX, KMIN |
test_sum_scheme.py |
direct, block and kahan on ill-conditioned sums |
test_batch.py, test_ranges.py |
batch dimensions and block-sparse ranges |
test_fp16.py |
FP16 against the oracle and the FP64 reference |
test_fast_math.py |
approximate exp/log within their error bounds, and actually active |
test_complex.py, test_ops.py, test_linalg.py |
complex formulas, PyKeOps formula operations, solve and linear operators |
Matrix: dimensions 1, 2, 3, 7, 16, 64, 65; K = 0, 1, 2, 31, 32, 33, 64, 128,
256, N−1, N, N+1; sizes at tile and block boundaries (1, 31, 32, 33, 255,
256, 257), multiples, primes (1031, 4099, 8191, 12289, 65537), M≪N, M≫N,
M = 0, N = 0; several i and j arguments, scalar and
vector parameters, vector outputs. The fast profile covers each dimension,
each K and each size family; the slow profile does the full
product (limited to M·N·D ≤ 8·10⁶ for the host reference).
Reference and tolerances¶
The formula is evaluated term by term in FP64 (evaluate_numpy, symox's NumPy
lambdify, floating-point constants passed as arguments: lambdify, like
SymPy, prints them with 15 significant digits) on the inputs rounded to the dtype, then reduced according to the
contract (NaN ignored, ties to the smallest j, order (value, j), padding
(+inf, −1)). For each term, an evaluation error bound T_ij:
T_ij = 0if an independent FP32 evaluation (Torch CPU oracle) reproduces the FP64 value exactly: the comparison is then exact, ties included (integer data of the matrix);- otherwise
T_ij = 4·(E_ij·s + C_ij) + 4·u·|F_ij| + 4·tiny, whereE_ijis the measured FP32/FP64 discrepancy,s = 1for an FP32 plan andu64/u32 = 2⁻²⁹for an FP64 plan (error proportional to the unit roundoff),C_ijthe largest FP64 deviation of the term when all inputs are perturbed by a factor1 ± u(4 fixed sign draws: conditioning, cancellations, function near a zero; a single FP32 evaluation may happen to be right by chance),uthe unit roundoff of the dtype andtinythe smallest normal (denormals, possible flush-to-zero); - a term whose class (finite, NaN, sign of infinity) differs between the two evaluations is indeterminate: its row is not judged.
Judgment per reduction:
| Reduction | Criterion |
|---|---|
| SUM | |r − ref| ≤ Σ_j T_ij + γ_N·L1 + 4·tiny, γ_N = N·u_acc/(1 − N·u_acc) (summation in any order), L1 = Σ_j |F_ij|; exact if all terms are exact, multiples of 1/2 and 2·L1 < 2⁻²/u_acc |
| COUNT | exact, except for terms whose predicate differs between FP32 and FP64 (at most one per ambiguous term) |
| MIN / MAX | exact if the row is exact or the extremum is infinite; otherwise |r − ref| ≤ max_j T_ij |
| ARGMIN / ARGMAX | exact (smallest j) if the row is exact; otherwise the returned index must be ε-optimal (F − T ≤ ref + T_ref), and exact if it is the only ε-optimal one; −1 only without candidate |
| KMIN / ARGKMIN | exact (+inf, −1) padding, sorted values; exact if the row is exact; otherwise values within max T, distinct indices, non-NaN, ε-equivalent rank by rank |
The dtypes, shapes and devices of the outputs are checked on every call.
Execution rules (test_execution.py)¶
- Non-default stream / queue: inputs written on a stream
sbehind ~40 2048² products (explicit data dependency), call undertorch.cuda.stream(s)/torch.xpu.stream(s), output consumed ons, synchronization ofsalone. Variant: output read on a second stream afterwait_stream. Sensitivity check (980 Ti): by forcing the launch on the default stream, both tests fail (zero inputs read). - Zero elements: M = 0, N = 0, M = N = 0 for the 9 reductions (identities of the contract: 0, +inf, −inf, −1, padding (+inf, −1)), K = 0 (shape (M, 0)), gradient of a SUM with N = 0 (zero gradients, shapes preserved).
- Destruction during execution:
delof theReductionand of the compiled object,gc.collect(), new compilation and new launch, while a 32768 × 131072 kernel is still queued (checked by an unfinished event); the result is compared with the FP64 reference on a sample of rows. The test reports whether the kernel was still running and whether the compiled object was collected (the backend's caches may keep it alive: allowed). - No host copy: second call (after compilation and selection),
tensor parameters or Python floats already cached. Two checks:
a
TorchDispatchModethat flags any ATen op mixing host and device or reading a scalar (_local_scalar_dense), on all backends; then the Torch profiler with device activity (CUDA, including under ROCm, orXPU) when it is available, without anyMemcpy HtoD/DtoHevent. A native copy outside Torch is only seen by the profiler. A new Python float for a parameter creates a device tensor (input H2D copy, see Concepts); outside the scope oflaunch. - Multi-GPU: run only if Torch sees at least two devices of the
same backend (outputs on the right device, reference, rejection of inputs on two
devices). Limitation: the test machines have no two cards from the
same vendor, and a PyTorch build sees only one vendor, so this test is
currently skipped, with this reason. Per-device placement (tensor's
device_index, current stream of that device) is only checked at index 0.
Fuzzing¶
_fuzz.Generator(seed) produces, reproducibly, trees over
x[c], y[c], p[c] and Python constants (integers, floats), with
+ − × /, unary minus, abs sin cos tanh atan exp log sqrt rsqrt square
clamp floor minimum maximum atan2, integer powers (2, 3, −1) and
rational powers (1/2, 3/2, −1/2), nested wheres and domains guarded by
where (log, sqrt, division), comparisons, eq/ne, & | ^ ~.
Reduction, K, axis, dtype, sizes and data (normal or integer) are drawn
together with the formula. On failure, the tree and the sizes are minimized then
recorded in tests/conformance/regressions/<backend>-seed<s>-….json
(frontend text, data regenerated from the seed); test_regression replays
each file on all available backends.
Known limitations¶
- The matrix accepts an explicit
UnsupportedErrorfor K > 256 and reports it as a warning; for K ≤ 256 a refusal is a failure. The three backends currently support K up to 1024. - The frontend brings the complex artifacts of the symbolic engine back to real
arithmetic (
builder.realify):I/zooin a branch value of aPiecewise(e.g.sqrt(where(c, a, -1.5))distributed over the branches) becomenan,re(a)→a,im(a)→ 0. Outside a branch, they remain rejected (UnsupportedError). - The reference and the judgment run on the host: long profile bounded in M·N·D.
- The FP64 error bound is derived from the scaled FP32 discrepancy; it is not a rigorous bound for every formula, but an estimate proportional to the measured conditioning, inflated by a factor of 4.