Skip to content

HIP backend (AMD ROCm)

This page explains how the hip backend compiles and launches its kernels, and why. Code: kernox/backends/hip/. Environment: Torch ROCm (torch 2.15.0+rocm10.0, index download.pytorch.org/whl/rocm10.0), no system ROCm required. Tests: tests/hip/.

Toolchain choice

hipRTC from Torch's ROCm package + hipModuleLoadData / hipModuleLaunchKernel from Torch's HIP runtime, bound through ctypes.

Rationale, established on a Radeon AI PRO R9700:

  • The official Torch ROCm wheels (2.15.0+rocm10.0) ship their own ROCm (_rocm_sdk_core: libamdhip64.so.7, libhiprtc.so.7, comgr, LLVM). Torch already loads libhiprtc into the process. Compiling with this hipRTC guarantees the same version as the runtime that will load the code object.
  • A system ROCm (for example 10.1, runtime 7.16) can be newer than Torch's (7.15). Loading /opt/rocm/lib/libamdhip64.so into the process would create a second HIP runtime whose contexts, streams and allocations are foreign to Torch's. hipcc --genco + hipModuleLoad would additionally require a system toolchain and one subprocess per compilation.
  • In-process compilation (~0.7 s per simple kernel, 3–11 s for KMIN K=128–256), with no dependency on the system ROCm: the venv is enough.

Opening the libraries: always RTLD_LOCAL (_hip._DLOPEN_MODE). Reopening with RTLD_GLOBAL a library already loaded by Torch promotes it, together with its dependencies (libamd_comgr → libLLVM.so.23.0git from _rocm_sdk_core), into the global scope. The static constructors of triton/_C/libtriton.so, which embeds its own LLVM with visible symbols, then bind to llvm::cl::Option::addArgument from libLLVM.so, which causes a SIGSEGV when importing torch._dynamo (torch.compile after Kernox). The crash is reproduced by a mere CDLL(libamdhip64 or libhiprtc, RTLD_GLOBAL), without launching any kernel. With RTLD_LOCAL, both orders and torch.compile of a real function pass (tests/hip/test_torch_compile_coexistence.py). The cost is zero: opening 2.7 ms, dynamo import 1.40 s versus 1.43 s without Kernox.

amdclang++ from the same package is only used for offline diagnostics (-Rpass-analysis=kernel-resource-usage, diagnostics.py); its VGPR counts agree with hipFuncGetAttribute on the hipRTC code object (checked for K=8).

Compilation options (all part of the cache key): --offload-arch=<gcnArchName> -O3 -std=c++17 -ffp-contract={on|off} -fno-fast-math -fno-gpu-flush-denormals-to-zero.

FMA contraction: -ffp-contract=on if plan.fp_contract (default, like CUDA/SYCL/PyKeOps; contraction within an expression only, never fast), -ffp-contract=off if plan.fp_contract=False (bit-for-bit reproducibility). Checked on R9700 (tests/hip/test_fp_contract.py): in FP64, off is bit-for-bit equal to the Torch reference without FMA, on differs from it in the last bit (≤ 1e-15); the two modes have distinct cache keys. The semantic tests of tests/hip/_cases.py use fp_contract=False (bit-for-bit references).

Measured limitation: on the pair code generated by symox, -ffp-contract=on produces no FMA. The square goes through the helper kp_pow2(v), so the product and the addition are in two distinct source expressions. The gfx1201 ISA is identical between on and off (gauss D16: 2 FMA FP32 / 13 FP64, all coming from exp; KNN D16: 0), gauss outputs bit-for-bit identical on 20,000² (FP32/FP64, D3/D16). With -ffp-contract=fast, which only does contraction (no fast-math), KNN D16 goes up to 15 FMA, like nvcc --fmad=true (15 FMA out of 16 on the CUDA side), and gauss to 57 FP32 / 29 FP64. In other words, the "within an expression" rule (on) is stricter here than nvcc's behaviour (fast). The backend uses on, as the contract requires.

Context, stream and lifetime rules

  1. A single HIP runtime: _hip.hip_runtime() opens the libamdhip64 already mapped by Torch (read from /proc/self/maps); same_runtime_as_torch() checks it (test).
  2. Device: modules are loaded and launched under torch.cuda.device(index); modules are cached by (key, device); no explicit context (primary context, like Torch).
  3. Stream: torch.cuda.current_stream(device).cuda_stream (a hipStream_t under ROCm). No host synchronization in launch (checked with torch.cuda.set_sync_debug_mode("error") and a stream still busy after launch returns).
  4. Dependencies: identical to native Torch operations: the caller orders its streams (wait_stream, events). Tested: inputs produced behind a sleep kernel on a non-default stream; producer on s1, kernel on s2 after wait_event.
  5. Buffer lifetime: the core allocates the outputs on the current stream; the inputs can be freed by Python right after launch: Torch's caching allocator only reuses them in stream order (tested with actual address reuse and overwriting). As for Torch operations, a tensor allocated on another stream and freed before the end of the kernel requires record_stream on the caller side.
  6. Errors: any HIP error on our side is consumed by hipGetLastError; otherwise Torch raises it at its next launch and attributes it to its own kernel (observed: hipErrorInvalidImage reported by a Torch addition after a corrupted code object).
  7. Destruction: modules are never unloaded (process cache): hipModuleUnload while a kernel is queued would be undefined, and unloading at interpreter exit can follow the destruction of the runtime. Destroying a HipBackend/HipCompiledReduction has no native effect.
  8. Zero elements: M = 0 and K = 0 launch nothing; N = 0 launches the kernel that writes the neutral values (no host-side filling).
  9. Multi-GPU: a single AMD GPU available for testing; not tested.

Compilation cache

$KERNOX_CACHE_DIR/hip (default ~/.cache/kernox/hip), <key>.co. Key = sha256 of {plan.key(), generator version, sha256 of the full source (pair function + kernel), compiler (hipRTC version, runtime version, path), options, arch}. Written to a temporary file in the same directory, fsync, then os.replace. One fcntl.flock lock per key (re-read under the lock: a single compilation for N processes, tested with 3). A code object unreadable by hipModuleLoadData → invalidation and recompilation (tested). In-memory cache by (key, device) on top.

Kernels and variants

A hipRTC module contains kp_reduce (grid (gx, S)) and kp_merge.

  • row traversal: one thread per row i, blocks of 256 threads (multiple of 32 and of 64: no assumption on the wavefront size, 32 on gfx12), tiles of TJ rows j in LDS (TJ = min(block, 32 KiB / row j)), i arguments in registers. Traversal in increasing j: "smallest index on ties" through strict comparisons. SUM: partial sum per tile then total. MIN/MAX: NaN ignored through strict comparison. ARGMIN/ARGMAX: v < best || (idx < 0 && v == best). KMIN: sorted list (value, int64 index) in registers, insertion by unrolled swaps.
  • j-splitting (jsplit): if the grid ceil(M / rows per block) has fewer than blocks_per_cu × CU blocks (32 × 32 by default, chosen from R9700 measurements), the j range is cut into S <= 64 ranges (multiples of TJ); each block (bx, s) writes its partial state into [S, M, W] (values + indices for ARG*/KMIN), allocated by Torch on the current device and stream. kp_merge merges in increasing order of s (= of j): sum, min/max, arg (the earlier partial wins on ties), K-way merge of the top-K lists in lexicographic order (value, j). Decision at launch (without recompilation); temporaries bounded by temp_max_bytes (256 MiB), published in CompileInfo.temp_bytes_per_launch (bound) and HipCompiledReduction.temp_bytes(M, N) / last_launch (exact).
  • Cooperative top-K (coop_topk): G lanes (G | 32) per row i; the list of G·L >= K elements is distributed (L ≈ 16 slots per lane, in registers); each lane evaluates a different j, the G candidates are broadcast by __shfl then inserted one by one (inter-lane shift through __shfl_up), full lexicographic comparison (value, j). Can be combined with jsplit.

Variants

Variant(name, params) with name ∈ {row, jsplit, coop_topk} and concrete parameters block, tile_j, split (off/auto), blocks_per_cu (if auto), group (coop). The variant is part of the cache key and is published in info.variant.

  • default_variant (ESTIMATE, deterministic): coop_topk if KMIN/ARGKMIN and K >= 32; j-splitting if the grid is below 32 × CU blocks and N > tile; block 256. Thresholds measured on an idle R9700.
  • variants(..., "measure") (<= 4, chosen for gfx1201): default; block 128 (wave32 occupancy); inverse splitting; then finer splitting (other blocks_per_cu: 8) or small tile (large D). Top-K: default, the other strategy (coop from K >= 16, or thread if K <= 128, otherwise coop G/2), coop 2G, inverse splitting.
  • variants(..., "patient") (<= 32): blocks 64/128/256/512 × tiles × splitting (off, auto 8, auto 32); top-K: groups with 4–32 slots per lane × blocks 128/256 × splitting, plus thread if K <= 128. Invalid variants excluded (block not a multiple of 64, LDS, more than 32 slots per lane).

Tests (tests/hip/test_variant_api.py, test_variants.py): all patient variants give identical results for MIN, ARGMAX, ARGMIN (massive ties, NaN, inf), COUNT, KMIN_ARGKMIN K = 8/70/130 and stay within the L1 tolerance for SUM (FP32/FP64), for (M, N) = (40, 700) and (700, 300); splittings forced up to S = 63; coop groups 2–32.

Measured resources (KMIN_ARGKMIN, d², D = 3, gfx1201)

python -m kernox.backends.hip.diagnostics thread,coop (per-kernel remarks + attributes of the loaded code object, in agreement). LDS 3072 B (FP32) / 6144 B (FP64), no SGPR spill.

strategy dtype K G VGPR VGPR spill scratch B/lane waves/SIMD
thread f32 8 1 53 0 0 16
thread f32 32 1 201 0 0 7
thread f32 64 1 256 69 156 5
thread f32 128 1 256 920 1504 5
thread f32 256 1 256 2696 4320 5
thread f64 32 1 207 0 0 7
thread f64 64 1 256 333 592 5
thread f64 128 1 256 1557 2280 5
thread f64 256 1 256 4149 6008 5
coop f32 32 2 101 0 0 12
coop f32 64 4 108 0 0 12
coop f32 128 8 130 0 0 10
coop f32 256 16 136 0 0 10
coop f64 32 2 111 0 0 12
coop f64 64 4 176 0 0 8
coop f64 128 8 167 0 0 9
coop f64 256 16 167 0 0 9

kp_merge: 11–17 VGPR; 272 B/lane of scratch (pointer array of the S partials, second pass only). The cooperative variant removes all the spills observed, at the cost of shuffles and sequential insertions.

K > 256: K_MAX = 1024, coop only (THREAD_K_MAX = 256). At most COOP_MAX_SLOTS = 32 slots per lane: G=16 up to K=512, G=32 beyond. No spill in kp_reduce (-Rpass-analysis, d², D=3):

dtype K G slots/lane VGPR spill occupancy
f32 512 16 32 213 0 7
f32 512 32 16 166 0 9
f32 1024 32 32 207 0 7
f64 512 16 32 226 0 6
f64 512 32 16 249 0 5
f64 1024 32 32 216 0 7

Tests: tests/hip/test_variants.py::test_coop_k_up_to_1024 (ties, NaN, K > N, j-splitting).

LOGSUMEXP and SUMSOFTMAXWEIGHT

thread strategy (the coop variant remains reserved for KMIN). State per row and per component (m, s[, t_1..t_E]) in kp_acc:

  • insertion of a term v: ignored if v == −inf; if v > m, c = exp(m − v), s = s·c + 1, t = t·c + g, m = v; otherwise q = (v == m) ? 1 : exp(v − m), s += q, t += q·g;
  • insertion level (variant parameter levels, HipOptions.soft_levels): 2 = tile state merged into the row state at the end of the tile (like SUM), 1 = direct insertion into the row state (half as many registers). Default: 1 with FP64 accumulation or if the state exceeds 256 B, 2 otherwise (measured on R9700: SSMW FP64 E64 189.7 → 100.9 ms, spill 852 → 140 B, ×0.60 versus Torch; FP32 below the threshold: no effect). The other level is included in measure/patient; j-splitting writes [S, M, W] (W = 2D for LSE, 2 + E for SSMW) and kp_merge merges with the same rule (factor exactly 1 when the maxima are equal, so an empty state (−inf, 0) is neutral and two +inf do not produce a spurious NaN);
  • m is never NaN; a NaN (term or weight) propagates into s/t;
  • result: LSE s NaN → NaN, m = +inf → +inf, otherwise m + log s (−inf if empty); SSMW m = +inf → NaN, otherwise t / s (0/0 if empty);
  • expf/exp, logf/log from the HIP libm, never __expf; FMA contraction according to plan.fp_contract.

Tests: tests/hip/test_softmax.py (forced splitting with empty slices, all patient variants, FP64 acc, fp_contract) and tests/conformance/test_softmax.py.

Summation schemes

HipBackend.sum_schemes = ("direct", "block", "kahan"). The schemes apply to SUM, and to s, t_e of LSE/SSMW. CompileInfo.options publishes sum_scheme=<scheme>, plus sum_block=T for block (T = tile_j).

  • direct: SUM accumulates each tile into a local accumulator, added to the total at the end of the tile (in effect, the same code as block). LSE/SSMW follow the levels rule above.
  • block: SUM, same code as direct, T = tile_j ≤ 256. LSE/SSMW: levels = 2 is enforced (fresh tile state merged), an explicit levels = 1 is refused. A splitting slice starts a tile; the last tile of a slice can be partial (≤ T).
  • kahan: Neumaier per component (kp_kadd), direct insertion without tiles (levels = 1 enforced for LSE/SSMW). The rescaling exp(m_old − m_new) multiplies s, c_s, t_e, c_te. The partials contain the compensations: SUM W = 2d, LSE 3D, SSMW 1 + 2d. kp_merge merges through TwoSum then adds c1 + c2. Result s + c if s is finite, otherwise s, with the test s − s == 0. Only additions and subtractions: fp_contract has no effect.

Tests: tests/hip/test_sum_scheme.py (forced splitting with empty slices, ±inf/NaN rows, Higham N = 2·10⁵ with splitting and fp_contract on/off, FP64 acc, enforced levels) and tests/conformance/test_sum_scheme.py.

Batch and block-sparse ranges

  • Batch (HipBackend.batch = True): batch index b0 + blockIdx.z, offset table boff (nargs, nb) in elements relative to data_ptr, outputs (nb, M, ·), partials (nb, S, M, W); z grid cut into chunks of GRID_Z_MAX = 65535. split_plan and the default variant count gx·nb blocks.
  • Ranges (HipBackend.ranges = True): kp_reduce_ranges, one block per work tile (i0, i1, k0, k1) of height ranges_tile_m = rows_per_block, reduction over ∪ red[k0:k1] without j-splitting; uncovered rows receive the neutral value through an N = 0 launch of kp_reduce. Ties go to the smallest j whatever the order of the intervals (comparison (value, j)). Batch + ranges: refused.

Tests: tests/hip/test_batch_ranges.py.

MIN_ARGMIN, MAX_ARGMAX, LOGSUMEXPWEIGHT

  • MIN_ARGMIN / MAX_ARGMAX: the ARGMIN / ARGMAX code (same pass, same (value, index) partials, same merge), which also writes value. Ties, NaN and neutral (±inf, −1) identical to MIN and ARGMIN.
  • LOGSUMEXPWEIGHT: state, insertion, levels, partials and summation schemes of SUMSOFTMAXWEIGHT; result m + log t_e with no special case (empty: −inf; t_e < 0: NaN; f = +inf: +inf if t_e > 0, NaN otherwise).

Tests: tests/hip/test_reductions_v08.py (forced splitting, ties, NaN, N = 0, block/kahan schemes, FP64 acc), the corresponding cases of tests/hip/test_batch_ranges.py, tests/conformance/test_reductions_v08.py.

FP16

  • Type kp_h (preamble HALF_PRELUDE of kernox/codegen/pair.py): wrapper around _Float16 in which every operation (+ − × ÷, comparisons) is computed and rounded in binary16. Conversions to kp_h are implicit (literals, integers; round to nearest), conversions from kp_h explicit. Thus a literal 1.5F or a call to expf never promotes the computation to FP32, which would happen with bare _Float16. Mathematical functions: computed in float then rounded to FP16, like Torch on CPU. A function without an FP16 overload is refused at generation (UnsupportedError, "dtype float16").
  • The pair function is generated by symox in FP32 (symox has no float16 dtype), then float is replaced by kp_h in it. The plan constants are already rounded to FP16 by the builder and are therefore printed exactly. A constant created by a symox simplification can undergo a double rounding (decimal → FP32 → FP16); this is rare.
  • Core: kp_t = kp_h; kp_acc = float by default (SUM, LOGSUMEXP families) or kp_h if acc_dtype="float16". In that case, kp_exp, kp_log and kp_nan are overloaded and sum_scheme="block" is the default. The new typedef kp_oacc carries the type of the accumulated outputs (plan.accumulated_output_dtype: FP16), with a final round-to-nearest conversion (±inf beyond 65504). Lane exchanges (__shfl, __shfl_up) go through float, which is exact. The partials are in acc_dtype; the LDS footprint is 2 B per component.
  • Evaluation order: the C and Torch printers order the terms of a symbolic sum differently (C: x0, x1, x2; Torch: x2, x0, x1). In FP16, a term can therefore differ by one ulp from the oracle. tests/conformance/test_fp16.py additionally checks that the error against the FP64 reference does not exceed twice that of the oracle.

Tests: tests/hip/test_fp16.py (bit-for-bit selections, including coop K = 64; SUM with FP32/FP16 acc with splitting; proof of FP16 computation through overflow; final ±inf conversion; LOGSUMEXP families; batch; ranges) and tests/conformance/test_fp16.py.