Skip to main content

nadam_step

Function nadam_step 

Source
pub fn nadam_step<T: Triton, const BLOCK_SIZE: i32>(
    params_ptr: T::Pointer<f32>,
    grad_ptr: T::Pointer<f32>,
    exp_avg_ptr: T::Pointer<f32>,
    exp_avg_sq_ptr: T::Pointer<f32>,
    n_elements: i32,
    lr: f32,
    beta1: f32,
    beta2: f32,
    eps: f32,
    weight_decay: f32,
    bias_corr2_sqrt: f32,
    coeff_g: f32,
    coeff_m: f32,
)
where T::I32Tensor: Tensor<i32, 1> + Comparison<i32, BoolTensor = T::BoolTensor>, T::Pointer<f32>: AddOffsets<i32, 1, T::I32Tensor, Output = T::Tensor<T::Pointer<f32>>>,
Expand description

NAdam step (Nesterov-accelerated Adam).

All bias correction terms and Nesterov coefficients are precomputed on the host and passed as scalars:

exp_avg    = beta1 * exp_avg    + (1 - beta1) * g
exp_avg_sq = beta2 * exp_avg_sq + (1 - beta2) * g²
denom      = sqrt(exp_avg_sq) / bias_corr2_sqrt + eps
p         -= lr * (coeff_g * g + coeff_m * exp_avg) / denom

Precomputed on host:

  • bias_corr2_sqrt = sqrt(1 - beta2^t)
  • coeff_g = (1 - mu_t) / (1 - mu_product) (grad contribution)
  • coeff_m = mu_t1 / (1 - mu_product_next) (moment contribution)

Grid: [ceil(n_elements / BLOCK_SIZE), 1, 1].