SigmoidSoftmaxTransformer

Sigmoid and softmax functions: how they work and when to use each

Sigmoid and softmax turn the scores of a classifier into probabilities. The sigmoid turns one score into a probability between 0 and 1. The softmax turns several scores, one per class, into probabilities that sum to 1. This article explains both with formulas and examples, and shows when to use each.

By Rahmatullah Musawi · · updated 9 October 2026 · 8 min read · sources checked 8 October 2026

Key points

  • The sigmoid function turns one score, which can be any number, into a probability between 0 and 1.[1]
  • The softmax function turns a list of scores into probabilities: each lies between 0 and 1, and together they sum to 1.[1]
  • The sigmoid answers a yes-or-no question: its output is the probability of yes, and 1 minus the output is the probability of no.[1] The softmax gives one probability per class. These sum to 1, so when one rises, at least one other must fall.[2]
  • Use one sigmoid to choose between two classes and the softmax to choose one class out of three or more.[1] When several classes can be correct at the same time, the softmax does not fit, and each class gets its own sigmoid.[3]
  • The transformer is the standard architecture for building large language models. It uses the softmax in two places. In attention, the softmax gives every word a weight. At the end of the model, it gives every word of the vocabulary a probability of coming next.[4] [5] [6]

A classifier assigns an input to a class. For a movie review, the classes could be positive and negative.1 To do this, the classifier first computes a score. A score can be any number, from far below zero to far above it.1 A probability, however, must lie between 0 and 1.1

Sigmoid and softmax are two functions that turn scores into probabilities. The sigmoid takes one score. The softmax takes several scores at once, one per class.1 Most examples in this article come from the textbook “Speech and Language Processing” by Jurafsky and Martin. Where a step was calculated for this article, the figure says so.

What is the sigmoid function?

The sigmoid function takes one number and turns it into a number between 0 and 1.1 It is named after its S shape and is also called the logistic function.1

Plotted as a curve, the sigmoid is almost a straight line near a score of 0. Far from 0, the curve flattens: large positive scores give outputs close to 1, and large negative scores give outputs close to 0.1 The output never reaches 0 or 1 exactly.7

With two classes, the output is the probability of one class, and the probability of the other class is 1 minus the output.1 The classifier chooses a class when the probability of that class is more than 0.5.1 The value 0.5 is called the decision boundary.1 The score that goes into the sigmoid is often called a logit.1

Example: the sigmoid turns a movie review's score of 0.833 into a probability of 0.70
A classifier decides whether a movie review is positive. From six properties of the review, such as the number of positive words, it computes the score = 0.833.1 The blue point on the curve is the output for this score. The column beside the curve is cut at the same height: the part below the cut is the probability of yes, and the part above it is the probability of no.
10.50−6−3036THE OUTPUTTHE SCORETHE BOUNDARYσ(0.833) = 0.70YES0.70NO0.30
  1. The formula, read “sigma of z”, is the output for the score . is Euler's number, about 2.718.
  2. The exampleThe textbook gives the output 0.70 for the score 0.833;1 the middle step was calculated for this article.
    z = 0.833→e−z = 0.435→1 / (1 + 0.435) = 0.70
  3. The decisionThe review is positive with a probability of 0.70 and negative with a probability of 1 − 0.70 = 0.30.1 Because 0.70 is more than the boundary of 0.5, the classifier decides that the review is positive.
A score of 0 gives exactly 0.5, because and 1 / (1 + 1) = 0.5. For any number , the sigmoid has the property .1 In the example, 0.30 is therefore the sigmoid of −0.833.

What is the softmax function?

The softmax function takes a list of scores, one per class, and turns them into probabilities. Each probability lies between 0 and 1, and together they sum to 1.1 The softmax is a generalization of the sigmoid to more than two classes;1 the next section shows how. The scores that go into a softmax are also called logits.1

The softmax works in three steps. First, it raises the number to the power of each score. The result is called the exponential of the score. Second, it adds up all the exponentials. Third, it divides each exponential by this sum. The results are the probabilities.1

Like the sigmoid, the softmax pushes values toward 0 or 1. When one score is larger than the others, its probability tends to be close to 1, and the probabilities of the smaller scores are small.1

Example: the softmax turns six scores into six probabilities that sum to 1
The textbook gives six scores and the six probabilities that the softmax calculates from them.1 The exponentials and the divisions were calculated for this article. Blue marks the largest score.
1.820.63.001.10.22−1.53.321.224.533.20.33−1.1THE EXPONENTIALSTHE SCORES÷ 33.230.050.090.010.10.740.01SIX PROBABILITIES, SUM 1
  1. Exponential, the exponential of the score , is Euler's number raised to the power . The result is always positive, even for a negative score: −1.5 becomes 0.22. The largest score, 3.2, gets by far the largest exponential.
  2. SumThe six exponentials add up to 33.23.
  3. DivisionEach exponential is divided by the sum. The formula of the softmax, below this note, says the same.1 For the score 3.2, the result is 24.53 / 33.23 = 0.74. The six results are the probabilities. In the drawing, they fill the column from 0 to 1 exactly.
The same calculation in numbers
Scores to probabilities6 scores
score divided by the sumprobability
0.61.821.82 / 33.230.05
1.13.003.00 / 33.230.09
−1.50.220.22 / 33.230.01
1.23.323.32 / 33.230.1
3.224.5324.53 / 33.230.74
−1.10.330.33 / 33.230.01
sum of the exponentials: 33.23sum of the probabilities: 1.00
In the formula, is the list of scores and is one of them. (In the sigmoid figure, was a single score.) is the number of scores, here 6. Both and count the scores from 1 to : names the score whose probability is calculated, and runs through all scores in the sum. The sign means “add up”. The scores and the rounded probabilities come from the textbook.1 The exponentials are rounded to two decimals; their sum, 33.23, was calculated before rounding.

Sigmoid and softmax: the differences

Both functions divide a total of 1, but among a different number of answers. The sigmoid divides it between two answers: its output is the probability of yes, and 1 minus the output is the probability of no.1 The softmax takes one score per class and gives one probability per class. These probabilities always sum to 1, so when one rises, at least one other must fall.2

The sigmoid divides 1 between two answers; the softmax divides it among all classes
The sigmoid takes one score and gives one output, the probability of yes; the probability of no is 1 minus the output. In the drawing of the softmax, a line connects every score to every output, because every output depends on all scores. The area of each circle shows a probability, on the same scale in both drawings. The numbers are those of the two previous figures.1
One sigmoidyes or no
0.833SCOREONE SIGMOIDyes 0.70no 0.301 − 0.70YES AND NO SUM TO 1
yes + no = 1.00the output depends on one score
One softmaxone of six classes
0.60.051.10.09−1.50.011.20.13.20.74−1.10.01SCORESONE SOFTMAXOUTPUTSTHE SIX OUTPUTS SUM TO 1
sum of the six: 1.00every output depends on all scores
With two classes, both functions give the same result
Softmax of two scoresone score fixed at 0
scoredivided by the sumoutput
0.8332.302.30 / 3.300.70
01.001.00 / 3.300.30
sigmoid of 0.833: 0.701 − 0.70 = 0.30

A sigmoid is a softmax with two classes in which one score is fixed at 0.2 Here, the score fixed at 0 is the score for “no”. For the movie review, the softmax of the scores 0.833 and 0 gives 0.70 for yes and 0.30 for no, the same values as the sigmoid. This was calculated for this article.

In both drawings the probabilities sum to 1: yes and no for the sigmoid,1 and the six classes for the softmax, which assumes that an input belongs to exactly one class.3 The formula shows the softmax of a score and a second score of 0. Dividing the top and the bottom of the fraction by , the exponential of , gives the sigmoid.

For the softmax, only the differences between the scores matter: adding the same number to every score does not change its outputs.2 The name “softmax” can be misleading: the function is a softened version of the arg max, not of the maximum.2 The arg max of a list says only which entry is the largest. The softmax gives the largest score the largest probability and gives the other scores smaller probabilities.

SigmoidSoftmax
InputOne score1A list of scores, one per class1
OutputOne value between 0 and 1, the probability of yes1One value between 0 and 1 for each class1
What sums to 1Yes and no1All classes1
One output depends onOne scoreAll scores2
Adding the same number to every scoreChanges the outputChanges nothing2
The question it answersIs it this class or not?1Which one of the classes is it?1

When to use each function

The choice depends on how many classes can be correct for one input. The softmax assumes that each input belongs to exactly one class.3 Its probabilities sum to 1, so two classes can never both get more than 0.5. When an input can belong to several classes at the same time, the softmax does not fit.3 Each class then gets its own sigmoid, which answers yes or no for that class only.3 1

A decision path: one sigmoid per class, softmax, or one sigmoid
Two questions lead to one of three choices. The rules and the examples come from the sources; the order of the questions was chosen for this article.
A CLASSIFICATION TASKCan more thanone class be correctat once?YesOne sigmoidper class1NoMore thantwo classes?YesSoftmaxfor all classes2NoOne sigmoidfor the two classes3
  1. Several classes can be correct: one sigmoid per classEach class gets its own sigmoid, and each sigmoid gives its own probability of yes. These probabilities do not have to sum to 1.3 Examples: a picture of a bowl that contains both apples and oranges;3 a text that is about religion, politics, finance and education at once.8 This task is called multilabel classification.8
  2. Exactly one of three or more classes: softmaxOnly one class is correct; an input cannot belong to several classes.1 Examples: a review that is positive, negative or neutral;1 a picture that shows an orange, an apple or a pear.8 This task is called multiclass classification.8
  3. Exactly one of two classes: one sigmoidThe sigmoid gives the probability of one class, and the probability of the other class is 1 minus this value.1 A softmax with two classes would give the same two values, so one sigmoid is enough. Examples: a review that is positive or negative;1 an email that is spam or not spam. For such an email, an output of 0.932 means that it is spam with a probability of 93.2%.7

Where and why transformers use the softmax

The transformer is the standard architecture for building large language models,4 which predict the next word of a text.1 It uses the softmax inside attention and at the end of the model.6 In both places, the model needs to turn a list of scores into numbers that sum to 1.

In attention, the model combines each word with information from the words around it.4 To do this for one word, it gives every word of the input a score.5 The softmax turns these scores into weights that are all positive and sum to 1.5 4 A weight says how much its word contributes to the output of attention.4

At the end of the model, choosing the next word is a classification in which every word of the vocabulary is a class.1 Exactly one word comes next, so the softmax fits, as in the decision path. The model first calculates one score, a logit, for every word of the vocabulary. It does this with a linear layer, a multiplication by numbers that the model learned during training. The softmax then turns the logits into probabilities.6 4

The softmax in a transformer: inside attention and at the end of the model
The softmax and the weighted sum are fixed calculations (grey mark); the numbers of the linear layer (blue mark) are learned by the model.
Inside attention
For one word, the softmax turns the scores of all words into weights.
SCORES, ONE PER WORDSoftmaxall positive, sum to 1WEIGHTS, ONE PER WORDWeighted sumof the value vectorsOUTPUT OF ATTENTION

This formula comes from the paper “Attention Is All You Need”, which proposed the transformer.6 Every word has three vectors (lists of numbers): a query, a key and a value.5 , and are the tables (matrices) that hold these vectors for all words. The score between two words is the dot product of the first word's query and the second word's key. A word also gets a score with itself.5 6 To calculate a dot product, multiply the two vectors entry by entry and add up the products. writes all these dot products at once; the means that the rows and columns of are swapped. The softmax turns the scores into weights, and multiplying by adds up the value vectors, each multiplied by its weight.5 In this formula, means the keys, not the number of scores.

is the number of entries in a query vector or a key vector; it is 64 in the paper.6 The scores are divided by because the authors suspected that, with a large , the dot products grow large and push the softmax into regions where its gradients are extremely small.6 The gradient guides the learning of the model. Where a function is almost flat, its gradient is very small.2

At the end of the model
Every word of the vocabulary has a score. The softmax turns each score into the probability that its word comes next.
VECTOR THE MODEL HAS COMPUTEDLinear layerone score per word of the vocabularySoftmaxall positive, sum to 1alltheyourthat1.20.90.1−0.50.440.330.150.08wordslogitsprobabilities

is the list of logits, one for each word of the vocabulary, and is the list of probabilities.4 In the formula of the softmax above, the list of scores was called . The example text is “So long and thanks for”. Of the four words that could follow, “all” has the largest logit and gets the largest probability, 0.44.4

The logits and probabilities come from the textbook's example, which uses a simplified vocabulary of only four words.4

Sources

  1. Logistic Regression and Text Classification (chapter 4) Jurafsky and Martin, Speech and Language Processing
  2. Deep Feedforward Networks (chapter 6) Goodfellow, Bengio and Courville, Deep Learning
  3. Neural networks: Multi-class classification Google, Machine Learning Crash Course
  4. Transformers and Pretraining (chapter 7) Jurafsky and Martin, Speech and Language Processing
  5. The Illustrated Transformer Jay Alammar
  6. Attention Is All You Need Vaswani et al., arXiv
  7. Logistic regression: Calculating a probability with the sigmoid function Google, Machine Learning Crash Course
  8. Multiclass and multioutput algorithms scikit-learn documentation

Written by Rahmatullah Musawi with the help of an AI assistant.