Module elu 

Structs

CeluBackward

Backward: dx = dy if x >= 0 else dy * exp(x/alpha)

CeluForward

Forward: y = max(0, x) + min(0, alpha*(exp(x/alpha) - 1))

CeluForwardDispatch

Forward: y = max(0, x) + min(0, alpha*(exp(x/alpha) - 1))

CeluOp

EluBackward

Backward: dx = dy if x > 0 else dy * alpha * exp(x)

EluForward

Forward: y = x if x > 0 else alpha*(exp(x) - 1)

EluForwardDispatch

Forward: y = x if x > 0 else alpha*(exp(x) - 1)

EluOp

SeluBackward

Backward: dx = SCALEdy if x > 0 else dy * SCALEALPHA*exp(x)

SeluForward

Forward: y = SCALE * (x if x > 0 else ALPHA*(exp(x) - 1))

SeluForwardDispatch

Forward: y = SCALE * (x if x > 0 else ALPHA*(exp(x) - 1))

SeluOp

Functions

celu_backward

Backward: dx = dy if x >= 0 else dy * exp(x/alpha)

celu_forward

Forward: y = max(0, x) + min(0, alpha*(exp(x/alpha) - 1))

elu_backward

Backward: dx = dy if x > 0 else dy * alpha * exp(x)

elu_forward

Forward: y = x if x > 0 else alpha*(exp(x) - 1)

selu_backward

Backward: dx = SCALEdy if x > 0 else dy * SCALEALPHA*exp(x)

selu_forward

Forward: y = SCALE * (x if x > 0 else ALPHA*(exp(x) - 1))