Module gelu 

Structs

GeluBackward

Backward of the tanh-GELU approximation. Let inner = x + ax³, s = sigmoid(2cinner), t = tanh(cinner) = 2s-1 d/dx = 0.5(1 + t) + x * 0.5 * sech²(cinner) * c(1+3a*x²)

GeluForward

Forward: y = x / (1 + exp(-2 * c * (x + a*x³))) where c = sqrt(2/pi), a = 0.044715 — the tanh GELU approximation.

GeluForwardDispatch

Forward: y = x / (1 + exp(-2 * c * (x + a*x³))) where c = sqrt(2/pi), a = 0.044715 — the tanh GELU approximation.

GeluOp

MishBackward

Backward: dx = dy * (tanh(sp) + x * (1 - tanh²(sp)) * sigmoid(x)) where sp = softplus(x). Recomputes all intermediates from x.

MishForward

Forward: y = x * tanh(softplus(x)) = x * tanh(log(1 + exp(x)))

MishForwardDispatch

Forward: y = x * tanh(softplus(x)) = x * tanh(log(1 + exp(x)))

MishOp

Functions

gelu_backward

Backward of the tanh-GELU approximation. Let inner = x + ax³, s = sigmoid(2cinner), t = tanh(cinner) = 2s-1 d/dx = 0.5(1 + t) + x * 0.5 * sech²(cinner) * c(1+3a*x²)

gelu_forward

Forward: y = x / (1 + exp(-2 * c * (x + a*x³))) where c = sqrt(2/pi), a = 0.044715 — the tanh GELU approximation.

mish_backward

Backward: dx = dy * (tanh(sp) + x * (1 - tanh²(sp)) * sigmoid(x)) where sp = softplus(x). Recomputes all intermediates from x.

mish_forward

Forward: y = x * tanh(softplus(x)) = x * tanh(log(1 + exp(x)))