HIP backend (AMD ROCm)¶
This page explains how the hip backend compiles and launches its kernels,
and why. Code: kernox/backends/hip/. Environment:
Torch ROCm (torch 2.15.0+rocm10.0, index download.pytorch.org/whl/rocm10.0),
no system ROCm required. Tests: tests/hip/.
Toolchain choice¶
hipRTC from Torch's ROCm package + hipModuleLoadData / hipModuleLaunchKernel
from Torch's HIP runtime, bound through ctypes.
Rationale, established on a Radeon AI PRO R9700:
- The official Torch ROCm wheels (2.15.0+rocm10.0) ship their own
ROCm (
_rocm_sdk_core:libamdhip64.so.7,libhiprtc.so.7, comgr, LLVM). Torch already loadslibhiprtcinto the process. Compiling with this hipRTC guarantees the same version as the runtime that will load the code object. - A system ROCm (for example 10.1, runtime 7.16) can be newer
than Torch's (7.15). Loading
/opt/rocm/lib/libamdhip64.sointo the process would create a second HIP runtime whose contexts, streams and allocations are foreign to Torch's.hipcc --genco+hipModuleLoadwould additionally require a system toolchain and one subprocess per compilation. - In-process compilation (~0.7 s per simple kernel, 3–11 s for KMIN K=128–256), with no dependency on the system ROCm: the venv is enough.
Opening the libraries: always RTLD_LOCAL (_hip._DLOPEN_MODE).
Reopening with RTLD_GLOBAL a library already loaded by Torch promotes it,
together with its dependencies (libamd_comgr → libLLVM.so.23.0git from
_rocm_sdk_core), into the global scope. The static constructors of
triton/_C/libtriton.so, which embeds its own LLVM with visible symbols, then
bind to llvm::cl::Option::addArgument from libLLVM.so, which causes a
SIGSEGV when importing torch._dynamo (torch.compile after Kernox). The
crash is reproduced by a mere CDLL(libamdhip64 or libhiprtc, RTLD_GLOBAL),
without launching any kernel. With RTLD_LOCAL, both orders and torch.compile of a
real function pass (tests/hip/test_torch_compile_coexistence.py). The cost
is zero: opening 2.7 ms, dynamo import 1.40 s versus 1.43 s without Kernox.
amdclang++ from the same package is only used for offline diagnostics
(-Rpass-analysis=kernel-resource-usage, diagnostics.py); its VGPR counts
agree with hipFuncGetAttribute on the hipRTC code object (checked for K=8).
Compilation options (all part of the cache key):
--offload-arch=<gcnArchName> -O3 -std=c++17 -ffp-contract={on|off} -fno-fast-math
-fno-gpu-flush-denormals-to-zero.
FMA contraction: -ffp-contract=on if plan.fp_contract
(default, like CUDA/SYCL/PyKeOps; contraction within an
expression only, never fast), -ffp-contract=off if
plan.fp_contract=False (bit-for-bit reproducibility). Checked on R9700
(tests/hip/test_fp_contract.py): in FP64, off is bit-for-bit equal to the
Torch reference without FMA, on differs from it in the last bit (≤ 1e-15); the two
modes have distinct cache keys. The semantic tests of
tests/hip/_cases.py use fp_contract=False (bit-for-bit references).
Measured limitation: on the pair code generated by symox,
-ffp-contract=on produces no FMA. The square goes through the helper
kp_pow2(v), so the product and the addition are in two distinct source
expressions. The gfx1201 ISA is identical between on and off (gauss D16: 2 FMA
FP32 / 13 FP64, all coming from exp; KNN D16: 0), gauss outputs
bit-for-bit identical on 20,000² (FP32/FP64, D3/D16). With
-ffp-contract=fast, which only does contraction (no fast-math),
KNN D16 goes up to 15 FMA, like nvcc --fmad=true (15 FMA out of
16 on the CUDA side), and gauss to 57 FP32 / 29 FP64. In other words, the
"within an expression" rule (on) is stricter here than nvcc's behaviour
(fast). The backend uses on, as the
contract requires.
Context, stream and lifetime rules¶
- A single HIP runtime:
_hip.hip_runtime()opens thelibamdhip64already mapped by Torch (read from/proc/self/maps);same_runtime_as_torch()checks it (test). - Device: modules are loaded and launched under
torch.cuda.device(index); modules are cached by(key, device); no explicit context (primary context, like Torch). - Stream:
torch.cuda.current_stream(device).cuda_stream(ahipStream_tunder ROCm). No host synchronization inlaunch(checked withtorch.cuda.set_sync_debug_mode("error")and a stream still busy afterlaunchreturns). - Dependencies: identical to native Torch operations: the caller
orders its streams (
wait_stream, events). Tested: inputs produced behind a sleep kernel on a non-default stream; producer on s1, kernel on s2 afterwait_event. - Buffer lifetime: the core allocates the outputs on the current
stream; the inputs can be freed by Python right after
launch: Torch's caching allocator only reuses them in stream order (tested with actual address reuse and overwriting). As for Torch operations, a tensor allocated on another stream and freed before the end of the kernel requiresrecord_streamon the caller side. - Errors: any HIP error on our side is consumed by
hipGetLastError; otherwise Torch raises it at its next launch and attributes it to its own kernel (observed:hipErrorInvalidImagereported by a Torch addition after a corrupted code object). - Destruction: modules are never unloaded (process cache):
hipModuleUnloadwhile a kernel is queued would be undefined, and unloading at interpreter exit can follow the destruction of the runtime. Destroying aHipBackend/HipCompiledReductionhas no native effect. - Zero elements: M = 0 and K = 0 launch nothing; N = 0 launches the kernel that writes the neutral values (no host-side filling).
- Multi-GPU: a single AMD GPU available for testing; not tested.
Compilation cache¶
$KERNOX_CACHE_DIR/hip (default ~/.cache/kernox/hip), <key>.co.
Key = sha256 of {plan.key(), generator version, sha256 of the full
source (pair function + kernel), compiler (hipRTC version, runtime
version, path), options, arch}. Written to a temporary file in the same
directory, fsync, then os.replace. One fcntl.flock lock per key
(re-read under the lock: a single compilation for N processes, tested with 3).
A code object unreadable by hipModuleLoadData → invalidation and recompilation
(tested). In-memory cache by (key, device) on top.
Kernels and variants¶
A hipRTC module contains kp_reduce (grid (gx, S)) and kp_merge.
rowtraversal: one thread per rowi, blocks of 256 threads (multiple of 32 and of 64: no assumption on the wavefront size, 32 on gfx12), tiles ofTJrowsjin LDS (TJ = min(block, 32 KiB / row j)),iarguments in registers. Traversal in increasingj: "smallest index on ties" through strict comparisons. SUM: partial sum per tile then total. MIN/MAX: NaN ignored through strict comparison. ARGMIN/ARGMAX:v < best || (idx < 0 && v == best). KMIN: sorted list (value, int64 index) in registers, insertion by unrolled swaps.- j-splitting (
jsplit): if the gridceil(M / rows per block)has fewer thanblocks_per_cu × CUblocks (32 × 32 by default, chosen from R9700 measurements), thejrange is cut intoS <= 64ranges (multiples ofTJ); each block(bx, s)writes its partial state into[S, M, W](values + indices for ARG*/KMIN), allocated by Torch on the current device and stream.kp_mergemerges in increasing order ofs(= ofj): sum, min/max, arg (the earlier partial wins on ties), K-way merge of the top-K lists in lexicographic order (value, j). Decision at launch (without recompilation); temporaries bounded bytemp_max_bytes(256 MiB), published inCompileInfo.temp_bytes_per_launch(bound) andHipCompiledReduction.temp_bytes(M, N)/last_launch(exact). - Cooperative top-K (
coop_topk): G lanes (G | 32) per rowi; the list ofG·L >= Kelements is distributed (L ≈ 16 slots per lane, in registers); each lane evaluates a differentj, the G candidates are broadcast by__shflthen inserted one by one (inter-lane shift through__shfl_up), full lexicographic comparison (value, j). Can be combined withjsplit.
Variants¶
Variant(name, params) with name ∈ {row, jsplit, coop_topk} and concrete
parameters block, tile_j, split (off/auto), blocks_per_cu (if auto),
group (coop). The variant is part of the cache key and is published in
info.variant.
default_variant(ESTIMATE, deterministic):coop_topkif KMIN/ARGKMIN and K >= 32; j-splitting if the grid is below32 × CUblocks and N > tile; block 256. Thresholds measured on an idle R9700.variants(..., "measure")(<= 4, chosen for gfx1201): default; block 128 (wave32 occupancy); inverse splitting; then finer splitting (otherblocks_per_cu: 8) or small tile (large D). Top-K: default, the other strategy (coop from K >= 16, or thread if K <= 128, otherwise coop G/2), coop 2G, inverse splitting.variants(..., "patient")(<= 32): blocks 64/128/256/512 × tiles × splitting (off, auto 8, auto 32); top-K: groups with 4–32 slots per lane × blocks 128/256 × splitting, plus thread if K <= 128. Invalid variants excluded (block not a multiple of 64, LDS, more than 32 slots per lane).
Tests (tests/hip/test_variant_api.py, test_variants.py): all
patient variants give identical results for MIN, ARGMAX,
ARGMIN (massive ties, NaN, inf), COUNT, KMIN_ARGKMIN K = 8/70/130 and
stay within the L1 tolerance for SUM (FP32/FP64), for (M, N) = (40, 700) and
(700, 300); splittings forced up to S = 63; coop groups 2–32.
Measured resources (KMIN_ARGKMIN, d², D = 3, gfx1201)¶
python -m kernox.backends.hip.diagnostics thread,coop (per-kernel
remarks + attributes of the loaded code object, in agreement). LDS 3072 B (FP32) /
6144 B (FP64), no SGPR spill.
| strategy | dtype | K | G | VGPR | VGPR spill | scratch B/lane | waves/SIMD |
|---|---|---|---|---|---|---|---|
| thread | f32 | 8 | 1 | 53 | 0 | 0 | 16 |
| thread | f32 | 32 | 1 | 201 | 0 | 0 | 7 |
| thread | f32 | 64 | 1 | 256 | 69 | 156 | 5 |
| thread | f32 | 128 | 1 | 256 | 920 | 1504 | 5 |
| thread | f32 | 256 | 1 | 256 | 2696 | 4320 | 5 |
| thread | f64 | 32 | 1 | 207 | 0 | 0 | 7 |
| thread | f64 | 64 | 1 | 256 | 333 | 592 | 5 |
| thread | f64 | 128 | 1 | 256 | 1557 | 2280 | 5 |
| thread | f64 | 256 | 1 | 256 | 4149 | 6008 | 5 |
| coop | f32 | 32 | 2 | 101 | 0 | 0 | 12 |
| coop | f32 | 64 | 4 | 108 | 0 | 0 | 12 |
| coop | f32 | 128 | 8 | 130 | 0 | 0 | 10 |
| coop | f32 | 256 | 16 | 136 | 0 | 0 | 10 |
| coop | f64 | 32 | 2 | 111 | 0 | 0 | 12 |
| coop | f64 | 64 | 4 | 176 | 0 | 0 | 8 |
| coop | f64 | 128 | 8 | 167 | 0 | 0 | 9 |
| coop | f64 | 256 | 16 | 167 | 0 | 0 | 9 |
kp_merge: 11–17 VGPR; 272 B/lane of scratch (pointer array of the S
partials, second pass only). The cooperative variant removes all the
spills observed, at the cost of shuffles and sequential insertions.
K > 256: K_MAX = 1024, coop only (THREAD_K_MAX = 256).
At most COOP_MAX_SLOTS = 32 slots per lane: G=16 up to K=512, G=32 beyond.
No spill in kp_reduce (-Rpass-analysis, d², D=3):
| dtype | K | G | slots/lane | VGPR | spill | occupancy |
|---|---|---|---|---|---|---|
| f32 | 512 | 16 | 32 | 213 | 0 | 7 |
| f32 | 512 | 32 | 16 | 166 | 0 | 9 |
| f32 | 1024 | 32 | 32 | 207 | 0 | 7 |
| f64 | 512 | 16 | 32 | 226 | 0 | 6 |
| f64 | 512 | 32 | 16 | 249 | 0 | 5 |
| f64 | 1024 | 32 | 32 | 216 | 0 | 7 |
Tests: tests/hip/test_variants.py::test_coop_k_up_to_1024 (ties, NaN,
K > N, j-splitting).
LOGSUMEXP and SUMSOFTMAXWEIGHT¶
thread strategy (the coop variant remains reserved for KMIN). State per row
and per component (m, s[, t_1..t_E]) in kp_acc:
- insertion of a term
v: ignored ifv == −inf; ifv > m,c = exp(m − v),s = s·c + 1,t = t·c + g,m = v; otherwiseq = (v == m) ? 1 : exp(v − m),s += q,t += q·g; - insertion level (variant parameter
levels,HipOptions.soft_levels):2= tile state merged into the row state at the end of the tile (like SUM),1= direct insertion into the row state (half as many registers). Default:1with FP64 accumulation or if the state exceeds 256 B,2otherwise (measured on R9700: SSMW FP64 E64 189.7 → 100.9 ms, spill 852 → 140 B, ×0.60 versus Torch; FP32 below the threshold: no effect). The other level is included inmeasure/patient; j-splitting writes[S, M, W](W = 2Dfor LSE,2 + Efor SSMW) andkp_mergemerges with the same rule (factor exactly 1 when the maxima are equal, so an empty state(−inf, 0)is neutral and two+infdo not produce a spurious NaN); mis never NaN; a NaN (term or weight) propagates intos/t;- result: LSE
s NaN → NaN,m = +inf → +inf, otherwisem + log s(−infif empty); SSMWm = +inf → NaN, otherwiset / s(0/0if empty); expf/exp,logf/logfrom the HIP libm, never__expf; FMA contraction according toplan.fp_contract.
Tests: tests/hip/test_softmax.py (forced splitting with empty slices,
all patient variants, FP64 acc, fp_contract) and
tests/conformance/test_softmax.py.
Summation schemes¶
HipBackend.sum_schemes = ("direct", "block", "kahan"). The schemes apply
to SUM, and to s, t_e of LSE/SSMW. CompileInfo.options publishes
sum_scheme=<scheme>, plus sum_block=T for block (T = tile_j).
direct: SUM accumulates each tile into a local accumulator, added to the total at the end of the tile (in effect, the same code as block). LSE/SSMW follow thelevelsrule above.block: SUM, same code as direct,T = tile_j≤ 256. LSE/SSMW:levels = 2is enforced (fresh tile state merged), an explicitlevels = 1is refused. A splitting slice starts a tile; the last tile of a slice can be partial (≤ T).kahan: Neumaier per component (kp_kadd), direct insertion without tiles (levels = 1enforced for LSE/SSMW). The rescalingexp(m_old − m_new)multipliess,c_s,t_e,c_te. The partials contain the compensations: SUMW = 2d, LSE3D, SSMW1 + 2d.kp_mergemerges through TwoSum then addsc1 + c2. Results + cifsis finite, otherwises, with the tests − s == 0. Only additions and subtractions:fp_contracthas no effect.
Tests: tests/hip/test_sum_scheme.py (forced splitting with empty slices,
±inf/NaN rows, Higham N = 2·10⁵ with splitting and fp_contract on/off,
FP64 acc, enforced levels) and tests/conformance/test_sum_scheme.py.
Batch and block-sparse ranges¶
- Batch (
HipBackend.batch = True): batch indexb0 + blockIdx.z, offset tableboff (nargs, nb)in elements relative todata_ptr, outputs(nb, M, ·), partials(nb, S, M, W); z grid cut into chunks ofGRID_Z_MAX = 65535.split_planand the default variant countgx·nbblocks. - Ranges (
HipBackend.ranges = True):kp_reduce_ranges, one block per work tile(i0, i1, k0, k1)of heightranges_tile_m = rows_per_block, reduction over∪ red[k0:k1]without j-splitting; uncovered rows receive the neutral value through anN = 0launch ofkp_reduce. Ties go to the smallest j whatever the order of the intervals (comparison(value, j)). Batch + ranges: refused.
Tests: tests/hip/test_batch_ranges.py.
MIN_ARGMIN, MAX_ARGMAX, LOGSUMEXPWEIGHT¶
- MIN_ARGMIN / MAX_ARGMAX: the ARGMIN / ARGMAX code (same pass, same
(value, index)partials, same merge), which also writesvalue. Ties, NaN and neutral(±inf, −1)identical to MIN and ARGMIN. - LOGSUMEXPWEIGHT: state, insertion, levels, partials and summation
schemes of SUMSOFTMAXWEIGHT; result
m + log t_ewith no special case (empty:−inf;t_e < 0: NaN;f = +inf:+infift_e > 0, NaN otherwise).
Tests: tests/hip/test_reductions_v08.py (forced splitting, ties, NaN,
N = 0, block/kahan schemes, FP64 acc), the corresponding cases of
tests/hip/test_batch_ranges.py, tests/conformance/test_reductions_v08.py.
FP16¶
- Type
kp_h(preambleHALF_PRELUDEofkernox/codegen/pair.py): wrapper around_Float16in which every operation (+ − × ÷, comparisons) is computed and rounded in binary16. Conversions tokp_hare implicit (literals, integers; round to nearest), conversions fromkp_hexplicit. Thus a literal1.5For a call toexpfnever promotes the computation to FP32, which would happen with bare_Float16. Mathematical functions: computed in float then rounded to FP16, like Torch on CPU. A function without an FP16 overload is refused at generation (UnsupportedError, "dtype float16"). - The pair function is generated by symox in FP32 (symox has no
float16 dtype), then
floatis replaced bykp_hin it. The plan constants are already rounded to FP16 by the builder and are therefore printed exactly. A constant created by a symox simplification can undergo a double rounding (decimal → FP32 → FP16); this is rare. - Core:
kp_t = kp_h;kp_acc = floatby default (SUM, LOGSUMEXP families) orkp_hifacc_dtype="float16". In that case,kp_exp,kp_logandkp_nanare overloaded andsum_scheme="block"is the default. The new typedefkp_oacccarries the type of the accumulated outputs (plan.accumulated_output_dtype: FP16), with a final round-to-nearest conversion (±inf beyond 65504). Lane exchanges (__shfl,__shfl_up) go through float, which is exact. The partials are inacc_dtype; the LDS footprint is 2 B per component. - Evaluation order: the C and Torch printers order the terms of a symbolic
sum differently (C:
x0, x1, x2; Torch:x2, x0, x1). In FP16, a term can therefore differ by one ulp from the oracle.tests/conformance/test_fp16.pyadditionally checks that the error against the FP64 reference does not exceed twice that of the oracle.
Tests: tests/hip/test_fp16.py (bit-for-bit selections, including coop
K = 64; SUM with FP32/FP16 acc with splitting; proof of FP16 computation through
overflow; final ±inf conversion; LOGSUMEXP families; batch; ranges)
and tests/conformance/test_fp16.py.