Introduction: The Sigmoid in Context
Welcome back! In our previous discussion, we explored backpropagation, and I hope you now have a clear understanding of what it is and how it works. Let's continue our deep learning journey.
Even with backpropagation applied to neural networks, certain challenges remain—most notably, the problem of vanishing gradients. While learning about this issue, I came across the sigmoid function, which plays a pivotal role in both logistic regression and neural network activation.
We all know the sigmoid function's equation: \( \sigma(x) = \frac{1}{1 + e^{-x}} \). It maps any real-valued number to a value between 0 and 1, making it ideal for probability estimation and as an activation function in neural networks. Yet, what often goes unnoticed is the story behind this equation. Where does it come from? Why does \( e \) (Euler's number) appear?
In this article, we'll trace the derivation of the sigmoid function from its mathematical roots to its modern applications, all while keeping an eye on the 2026 AI landscape where activation functions have become even more specialized.
How Do We Actually Use Sigmoid?
Before diving into its derivation, let's recap how sigmoid is used in practice. In logistic regression, we apply the sigmoid function to a linear combination of inputs to obtain a probability score. As an activation function in neural networks, it introduces non-linearity, enabling the network to learn complex patterns. Despite its historical importance, recent years have seen a shift toward alternatives like ReLU and its variants to combat vanishing gradients, but sigmoid remains foundational—especially in output layers for binary classification and in attention mechanisms (e.g., in some 2026 transformer architectures).
The Mathematical Origins: From Odds to Probabilities
The sigmoid function didn't emerge out of thin air. It's deeply rooted in probability theory. Let's start with the concept of odds. In statistics, odds represent the ratio of the probability of an event occurring to the probability of it not occurring: \( \text{odds} = \frac{p}{1-p} \), where \( p \) is the probability.
Taking the natural logarithm of the odds (log-odds or logit function), we get: \( \log\left(\frac{p}{1-p}\right) \). This transformation maps probabilities (0 to 1) to the real number line \((-\infty, \infty)\).
Now, the sigmoid is the inverse of the logit function. If we set \( x = \log\left(\frac{p}{1-p}\right) \), we can solve for \( p \) in terms of \( x \):
\( e^x = \frac{p}{1-p} \) \( e^x (1-p) = p \) \( e^x - e^x p = p \) \( e^x = p(1 + e^x) \) \( p = \frac{e^x}{1 + e^x} \)
By dividing both numerator and denominator by \( e^x \), we arrive at the familiar form:
\( \sigma(x) = \frac{1}{1 + e^{-x}} \)
This derivation shows that the sigmoid is not arbitrary—it emerges naturally from modeling log-odds, a concept that predates modern neural networks and remains central to logistic regression.
Why Does \( e \) Appear?
The constant \( e \approx 2.71828 \) appears because of its unique property: the derivative of \( e^x \) is itself. This property simplifies calculus in many ways, and in the context of the sigmoid, it leads to a beautiful derivative: \( \sigma'(x) = \sigma(x) \cdot (1 - \sigma(x)) \). This elegant form is computationally efficient and interpretable: the gradient is maximized at \( x = 0 \) (where \( \sigma = 0.5 \)) and diminishes as \( x \) moves away—hence the vanishing gradient problem in deep networks.
Sigmoid in Neural Networks: Then and Now (2026 Perspective)
In the early days of deep learning, sigmoid was the default activation function. However, its saturating nature—where gradients become vanishingly small for large inputs—led to the adoption of ReLU and its variants (e.g., Swish, GELU). As of 2026, sigmoid is still used in specific roles:
- Binary classification output layers: Producing probabilities between 0 and 1.
- Gating mechanisms: In LSTM and GRU units, sigmoid gates control information flow.
- Attention mechanisms: Some 2026 transformer architectures use sigmoid-based attention to weight feature interactions.
Conclusion: A Function of Enduring Relevance
The sigmoid function is more than just an equation—it's a bridge between probability theory and neural computation. By understanding its derivation, we gain a deeper appreciation for its role and limitations. While newer activation functions have emerged, the sigmoid's influence persists, reminding us that mathematical elegance often underpins practical success.