Module misc

Structs§

LeakyReluBackward: Backward: dx = dy if x > 0 else negative_slope * dy
LeakyReluForward: Forward: y = x if x > 0 else negative_slope * x
LeakyReluForwardDispatch: Forward: y = x if x > 0 else negative_slope * x
LeakyReluOp
SoftplusBackward: Backward: dx = dy * sigmoid(betax) For betax > threshold: dx ≈ dy
SoftplusForward: Forward: y = (1/beta) * log(1 + exp(betax)) For betax > threshold: y ≈ x (numerically safe pass-through)
SoftplusForwardDispatch: Forward: y = (1/beta) * log(1 + exp(betax)) For betax > threshold: y ≈ x (numerically safe pass-through)
SoftplusOp
SoftshrinkBackward: Backward: dx = dy if |x| > lambda else 0
SoftshrinkForward: Forward: y = x - lambda if x > lambda, x + lambda if x < -lambda, else 0
SoftshrinkForwardDispatch: Forward: y = x - lambda if x > lambda, x + lambda if x < -lambda, else 0
SoftshrinkOp
SoftsignBackward: Backward: dx = dy / (1 + |x|)²
SoftsignForward: Forward: y = x / (1 + |x|)
SoftsignForwardDispatch: Forward: y = x / (1 + |x|)
SoftsignOp
ThresholdBackward: Backward: dx = dy if x > threshold else 0
ThresholdForward: Forward: y = x if x > threshold else value
ThresholdForwardDispatch: Forward: y = x if x > threshold else value
ThresholdOp

leaky_relu_backward: Backward: dx = dy if x > 0 else negative_slope * dy
leaky_relu_forward: Forward: y = x if x > 0 else negative_slope * x
softplus_backward: Backward: dx = dy * sigmoid(betax) For betax > threshold: dx ≈ dy
softplus_forward: Forward: y = (1/beta) * log(1 + exp(betax)) For betax > threshold: y ≈ x (numerically safe pass-through)
softshrink_backward: Backward: dx = dy if |x| > lambda else 0
softshrink_forward: Forward: y = x - lambda if x > lambda, x + lambda if x < -lambda, else 0
softsign_backward: Backward: dx = dy / (1 + |x|)²
softsign_forward: Forward: y = x / (1 + |x|)
threshold_backward: Backward: dx = dy if x > threshold else 0
threshold_forward: Forward: y = x if x > threshold else value