Module misc 

Other activations that don’t fit the other modules.

Structs

LeakyRelu

Leaky ReLU activation layer: x for x >= 0, negative_slope * x otherwise.

Softplus

Softplus activation layer: a smooth ReLU approximation, (1/beta) * ln(1 + e^(beta*x)), reverting to linear above threshold for numerical stability.

Softshrink

Softshrink activation layer: shrinks values toward zero by lambda.

Softsign

Softsign activation layer: x / (1 + |x|).

Threshold

Threshold activation layer: passes x through if x > threshold, else outputs value.