arXiv 2507.22090
Hybrid activation functions for deep neural networks: S3 and S4 -- a novel approach to gradient flow optimization
By Sergii Kavun
Published 2025-07-29
Wiki summary
Explore the paper's summary, context, and related research on Papiers.
Activation functions are critical components in deep neural networks, directly influencing gradient flow, training stability, and model performance. Traditional functions like ReLU suffer from dead neuron problems, while sigmoid and tanh exhibit vanishing gradient issues. We introduce two novel hybrid activation functions: S3 (Sigmoid-Softsign) and its improved version S4 (smoothed S3). S3 combines sigmoid for negat…