Welcome to 
Math in AI!
I’m Betty Alpha.
New series,
new accessories!

Excellent. I came
for viral AI
content.

Alpha and beta,
tucked right into my
hairpin. Think
“alpha-beta”… Betty
Alpha!

A wearable 
memory aid.
I love it.

And
formula-embroidered
pumps! A little pi,
a little infinity, 
a little integral.

Are those
limited
edition?

Mathematically
fabulous edition.

I study
biology, 
but I love 
seeing ideas
connect
across
subjects.

Curiosity is
always
welcome
here!

Sure, but 
math is boring. 
And difficult.
Can we skip
to the AI magic?

DIFFICULT IS
ALLOWED! 
“BORING” BEFORE
YOU EVEN TRY
IS NOT!

I appear
to have
activated
teacher
rage mode.

Struggling is fine.
Questions are excellent.
Dismissing yourself before you
start? Absolutely not.

Okay. 
One honest
try.

That’s all I ask.

Meet our special
assistant teacher:
Fullstack Carrot!

I’ll connect the
math to the
computer stuff. 
No coding experience
required.

A vegetable with
better
qualifications than
me.

Imagine an AI
judging three
possible answers. 
It produces scores
called logits.

Let’s use apple,
banana, and pear.
Their scores are 2,
1, and 0.

Those aren’t 
probabilities yet?

2

1

0

Right. Scores can be
positive, zero, or
negative. They don’t
have to add up to
one.

Can’t we just divide
each score by the
total?

Not reliably:
negative scores
could make negative
probabilities, and
the total could be
zero.

Softmax does two
things: exponentiate
each score, then
divide by the total.

\[
p_i=\operatorname{softmax}(z)_i=\frac{e^{z_i}}{\sum_j
e^{z_j}} \]

Here, zᵢ is one
score. The sum in
the denominator
includes every
candidate.

The exponential eˣ
is always
positive—even when x
is negative.

For scores 2, 1, 0,
the transformed
amounts are about
7.39, 2.72, and 1.

So zero gets e⁰ = 1.
Not zero!

Now divide each
amount by their
total, about 11.11.

Apple: 66.5%.
Banana: 24.5%. 
Pear: 9.0%.

All positive, 
and together Malus,
Musa, and Pyrus
make 100%.

Malus

Musa

Pyrus

Biology terms
Malus = Apple
Musa = Banana
Pyrus = Pear

So softmax
chooses
apple?

Softmax gives
weights. A separate
rule can choose the
biggest weight—or
randomly sample
using those weights.

It preserves which
score is largest. It
doesn’t make the
choice by itself.

Does a 66.5% output
mean the AI is
actually right 66.5%
of the time?

Not automatically.
These are the
model’s
probabilities, not a
guarantee of
correctness.

To trust that
interpretation, we
need to check how
well its confidence
matches real
outcomes.

But why this
formula? Why not one
of infinitely many
others?

Excellent question.
Softmax isn’t the
only possible rule.
It has a remarkably
useful combination
of properties.

It handles any real
scores, makes
positive weights
that sum to one, and
changes smoothly.

Smooth changes
help training algorithms
adjust the model.

Higher scores get
higher weights,
without an abrupt
winner-takes-all
jump.

Only score
differences matter.
Add the same number
to every score, and
the probabilities
stay unchanged.

Scores 2, 1, 0 
and 102, 101, 100
produce the same
softmax.

So giving everyone
bonus points changes
nobody’s share.

Here’s the
exponential’s
special trick: a
score difference
becomes a
probability ratio.

\[
\frac{p_i}{p_j}=e^{z_i-z_j}
\]

A one-point lead
means about 2.72
times the weight.
The same gap has the
same meaning
wherever the scores
start.

Is there a deeper
reason than “nice
properties”?

Yes! Entropy
measures 
how spread
out uncertainty is.

Among distributions
with a specified
average score,
maximizing entropy
gives an exponential
form. Softmax is
that form, with the
score scale set
appropriately.

So it can avoid
adding extra
certainty beyond the
constraint?

Exactly. A beautiful
statistical
connection—not proof
that every AI must
use it.

Its mathematical
family is much older
than AI. Statistical
physics used
exponential
probabilities in the
Boltzmann–Gibbs
distribution.

There, lower energy
gets more weight;
here, higher score
gets more weight.

Right—the sign
changes with the
meaning of the
quantity.

Related ideas
appear in
statistics and
choice models,
including Luce’s
1959 work on choice.

John Bridle helped
introduce the name
“softmax” into
neural-network work
around 1989.

So AI borrowed a
classic recipe and
gave it a new
kitchen.

With only two
choices, softmax
connects directly to
the logistic
sigmoid.

\[
p_1=\frac{1}{1+e^{-(z_1-z_2)}}
\]

The probability
depends on the
difference between
the two scores. Same
family!

Temperature
lets us adjust how
concentrated the
weights are.

\[
p_i=\frac{e^{z_i/T}}{\sum_j
e^{z_j/T}},\qquad
T>0 \]

Lower T favors the
largest score more
strongly. Higher T
spreads the weights
more evenly.

Not actual heat.
Got it.

One practical
trick: subtract the
largest score before
computing the
exponentials.

\[ m=\max_j
z_j,\qquad
p_i=\frac{e^{z_i-m}}{\sum_j
e^{z_j-m}} \]

Same probabilities,
but the exponentials
cannot explode into
enormous numbers.

A math trick that
prevents computer
drama. Respect.

Now let’s turn
choosing into
connecting. This is
where transformers
use softmax.

Could we use a
Chloris sentence?

Try: “The leaf
absorbs sunlight.”
Each word becomes a
token representation—
simplified for
our example.

Chloris

Of course you
named the leaf.

Each token
gets learned
numerical vectors:
a query Q, a key K, and 
a value V.

Think: 
query = what
I’m looking for; 
key = what 
can match;
value = information
to share.

That’s an
analogy. The actual
objects are learned
arrays of numbers, not
written questions.

For “absorbs,”
compare its query
with the keys of the
available tokens.

A dot product
multiplies matching
components and adds
them. It creates a
compatibility score.

The model 
learns those
representations from
training—not from us
hand-writing a rule
about Chloris.

Chloris

More components can
make dot products
vary more widely.
Attention divides by
the square root of
the key dimension.

\[
s_{ij}=\frac{q_i\cdot
k_j}{\sqrt{d_k}} \]

Under a 
simple
independent, 
zero-mean,
unit-variance model,
dot-product variance
grows like dₖ. This
scaling controls
that growth.

It helps keep
softmax from
becoming too sharply
concentrated just
because vectors are
large.

For an
invented
illustration, say
the scaled scores
for “leaf,”
“absorbs,” and
“sunlight” are 2, 1, 0.

Softmax gives that
query the same
66.5%, 24.5%, and
9.0% weights we
calculated earlier.

Illustrative
numbers—not a claim
about what a real
trained model would
attend to.

Then use those
weights to blend the
value vectors. This
is a weighted sum,
not just picking one
word.

\[ o_i=\sum_j
p_{ij}v_j \]

The result is a
new
representation
enriched by
information
from
other tokens.

Put those steps
together, and you
get scaled
dot-product
attention.

\[
\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^{\mathsf
T}}{\sqrt{d_k}}\right)V
\]

Compare 
queries with
keys. Scale the
scores. Apply
softmax across each
query’s allowed
keys. Blend the
values.

One formula, four
jobs. I can actually
follow that.

For many
text-generating
transformers, a
causal mask blocks
attention to future
tokens.

Softmax then shares
weight only among
the positions that
are allowed.

So attention depends
on both the learned
scores and which
connections are
permitted.

Multiple
attention heads
can learn
different kinds of
relationships.

But attention is one
part of a
transformer.
Feed-forward layers,
position
information, and
other components
matter too.

Not the whole AI
squeezed into a
single equation.
Understood.

Softmax doesn’t
magically choose. It
turns scores into
weights—and
attention uses those
weights to mix
information.

Yes!

And maybe 
math is…
viral-worthy?

Careful. 
You’re
dangerously 
close to
genuine 
curiosity.

A formula 
from old
physics helps
modern
computers
connect
words. That’s
wonderful.

That’s Math in AI:
small ideas doing
astonishing work.

Next time, 
we can explore
how models
learn from their
mistakes.

I’ll come
back.
Voluntarily.

Excellent.
Curiosity
looks good
on
everyone.

Appendix