KKKZOZ's Blog
Post / 2026-03-18
kkkzoz

Probability and Statistics

Math
Doc no.
KKZ-TN-154
Date
2026-03-18
GMT day
2026:077
Est. read
3 min

01L1 范数#

The norm calculates the sum of the absolute values of all elements within a vector or matrix.

Calculation: For a weight vector with elements, the norm is defined as:

02L2 范数#

The norm calculates the standard Euclidean distance of the weight vector from the origin. It is the square root of the sum of the squared values of the elements.

Calculation: For the same weight vector , the norm is defined as:

03KL 散度#

KL Divergence calculates the expected excess surprise from using an approximation distribution instead of the true distribution.

Calculation: For two discrete probability distributions, (the true distribution) and (the approximated or predicted distribution), defined on the same probability space , the KL Divergence from to is defined as:

Mathematical Breakdown:

  • represents the actual probability of an event occurring.
  • represents the model's predicted probability of that same event .
  • The ratio compares these probabilities. If they are identical, the ratio is 1, and the becomes 0, meaning there is zero divergence.
  • We multiply the log value by to compute the expected value (the weighted average) across the entire true distribution.

3.1Application in ML#

While norms are typically used to penalize or prune network weights, KL Divergence is predominantly used to optimize network outputs that represent probabilities.

  • Knowledge Distillation

    • When compressing a massive neural network (the "teacher") into a smaller network (the "student"), KL divergence is used as the loss function.
    • The student model is trained to minimize the KL divergence between its output probability distribution () and the teacher's output probability distribution (), forcing the student to mimic the teacher's exact confidence levels across all classes.

  • Cross-Entropy Loss

    • In standard classification tasks, minimizing the Cross-Entropy loss is mathematically equivalent to minimizing the KL Divergence between the training data's one-hot encoded labels () and the model's softmax predictions ().

3.2Calculation Example#

  • Vector (The True Distribution): This represents the actual, ground-truth probabilities.

  • Vector (The Predicted Distribution): This represents what our neural network predicts the probabilities are.

  • Element 1:

  • Element 2:

  • Element 3:

  • Element 4:

Finally, we sum these individual components together:

Comments