arXiv 2507.22090
Hybrid activation functions for deep neural networks: S3 and S4 -- a novel approach to gradient flow optimization
By Sergii Kavun
Published 2025-07-29
Mindmap
Browse the paper's core ideas, clusters, and relationships in a structured outline.
Activation functions are critical components in deep neural networks, directly influencing gradient flow, training stability, and model performance. Traditional functions like ReLU suffer from dead neuron problems, while sigmoid and tanh exhibit vanishing gradient issues. We introduce two novel hybrid activation functions: S3 (Sigmoid-Softsign) and its improved version S4 (smoothed S3). S3 combines sigmoid for negat…