Summary of PyTorch Loss Functions

Summary of PyTorch Loss Functions

by
Summary of PyTorch Loss Functions
Source: Pythonic Biologist

This article is about 1900 words long, and it is recommended to read it in 8 minutes.
TensorFlow and PyTorch are quite similar; this article introduces loss functions using PyTorch as an example.

19 Types of Loss Functions

1. L1 Loss L1Loss

Calculates the absolute difference between output and target.

torch.nn.L1Loss(reduction='mean')

Parameters:

reduction – three values: none (no reduction), mean (returns the average of losses), sum (returns the sum of losses). Default: mean.
2. Mean Squared Error Loss MSELoss
Calculates the mean squared difference between output and target.
torch.nn.MSELoss(reduction='mean')
Parameters:
reduction – three values: none (no reduction), mean (returns the average of losses), sum (returns the sum of losses). Default: mean.
3. Cross Entropy Loss CrossEntropyLoss
Effective for training classification problems with C categories. The optional parameter weight must be a 1D Tensor, and weights will be assigned to each category. Very effective for unbalanced training sets.
In multi-class tasks, softmax activation function + cross-entropy loss function is often used because cross-entropy describes the difference between two probability distributions, while the output of the neural network is a vector, not in the form of a probability distribution. Therefore, the softmax activation function is needed to “normalize” a vector into the form of a probability distribution, and then the cross-entropy loss function is used to calculate the loss.
Summary of PyTorch Loss Functions
torch.nn.CrossEntropyLoss(weight=None, ignore_index=-100, reduction='mean')
Parameters:
weight (Tensor, optional) – custom weights for each category. Must be a Tensor of length C.
ignore_index (int, optional) – sets a target value that will be ignored and will not affect the gradient of the input.
reduction – three values: none (no reduction), mean (returns the average of losses), sum (returns the sum of losses). Default: mean.
4. KL Divergence Loss KLDivLoss
Calculates the KL divergence between input and target. KL divergence can be used to measure the distance between different continuous distributions, and is very effective when performing direct regression on the space of continuous output distributions (discrete sampling).
torch.nn.KLDivLoss(reduction='mean')
Parameters:
reduction – three values: none (no reduction), mean (returns the average of losses), sum (returns the sum of losses). Default: mean.
5. Binary Cross Entropy Loss BCELoss
Cross-entropy calculation function for binary classification tasks. Used to measure reconstruction error, such as in autoencoders. Note that the target value t[i] must be in the range of 0 to 1.
torch.nn.BCELoss(weight=None, reduction='mean')
Parameters:
weight (Tensor, optional) – custom weights for each batch element’s loss. Must be a Tensor of length “nbatch”.
6. BCEWithLogitsLoss
The BCEWithLogitsLoss function integrates the Sigmoid layer into the BCELoss class. This version is numerically more stable than using a simple Sigmoid layer and BCELoss, as combining these two operations into one layer allows for numerical stability using the log-sum-exp trick.
torch.nn.BCEWithLogitsLoss(weight=None, reduction='mean', pos_weight=None)
Parameters:
weight (Tensor, optional) – custom weights for each batch element’s loss. Must be a Tensor of length “nbatch”.

7. Margin Ranking Loss

torch.nn.MarginRankingLoss(margin=0.0, reduction='mean')
For each instance in a mini-batch, the loss function is as follows:Summary of PyTorch Loss Functions
Parameters:
margin: default value 0

8. Hinge Embedding Loss

torch.nn.HingeEmbeddingLoss(margin=1.0, reduction='mean')

For each instance in a mini-batch, the loss function is as follows:Summary of PyTorch Loss Functions

Parameters:

margin: default value 1

9. Multi-Label Classification Loss MultiLabelMarginLoss

torch.nn.MultiLabelMarginLoss(reduction='mean')
For each sample in a mini-batch, the loss is calculated as follows:Summary of PyTorch Loss Functions

10. Smooth L1 Loss SmoothL1Loss

Also known as Huber loss function.
torch.nn.SmoothL1Loss(reduction='mean')

Summary of PyTorch Loss Functions

Where:

Summary of PyTorch Loss Functions

11. Logistic Loss for 2-Class SoftMarginLoss

torch.nn.SoftMarginLoss(reduction='mean')
Summary of PyTorch Loss Functions

12. Multi-Label One-Versus-All Loss MultiLabelSoftMarginLoss

torch.nn.MultiLabelSoftMarginLoss(weight=None, reduction='mean')
Summary of PyTorch Loss Functions

13. Cosine Loss CosineEmbeddingLoss

torch.nn.CosineEmbeddingLoss(margin=0.0, reduction='mean')

Summary of PyTorch Loss Functions

Parameters:
margin: default value 0

14. Hinge Loss for Multi-Class MultiMarginLoss

torch.nn.MultiMarginLoss(p=1, margin=1.0, weight=None, reduction='mean')

Summary of PyTorch Loss Functions

Parameters:
p=1 or 2 default value: 1 margin: default value 1

15. Triplet Loss TripletMarginLoss

Similar to Siamese networks, a specific example: given A, then given B and C, see who is more similar to A between B and C.
Summary of PyTorch Loss Functions
torch.nn.TripletMarginLoss(margin=1.0, p=2.0, eps=1e-06, swap=False, reduction='mean')

Summary of PyTorch Loss Functions

Where:

Summary of PyTorch Loss Functions

16. Connectionist Temporal Classification Loss CTCLoss
CTC connectionist temporal classification loss can automatically align data that is not aligned, mainly used for training serialized data without prior alignment. For example, speech recognition, OCR recognition, etc.
torch.nn.CTCLoss(blank=0, reduction='mean')
Parameters:
reduction – three values: none (no reduction), mean (returns the average of losses), sum (returns the sum of losses). Default: mean.
17. Negative Log Likelihood Loss NLLLoss
Negative log likelihood loss. Used for training classification problems with C categories.
torch.nn.NLLLoss(weight=None, ignore_index=-100, reduction='mean')
Parameters:
weight (Tensor, optional) – custom weights for each category. Must be a Tensor of length C.
ignore_index (int, optional) – sets a target value that will be ignored and will not affect the gradient of the input.
18. NLLLoss2d
Negative log likelihood loss for image inputs. It calculates the negative log likelihood loss for each pixel.
torch.nn.NLLLoss2d(weight=None, ignore_index=-100, reduction='mean')
Parameters:
weight (Tensor, optional) – custom weights for each category. Must be a Tensor of length C.
reduction – three values: none (no reduction), mean (returns the average of losses), sum (returns the sum of losses). Default: mean.
19. Poisson NLL Loss
The negative log likelihood loss for Poisson distribution:
torch.nn.PoissonNLLLoss(log_input=True, full=False, eps=1e-08, reduction='mean')
Parameters:
log_input (bool, optional) – If set to True, the loss will be computed according to the formula exp(input) – target * input; if set to False, the loss will be computed as input – target * log(input+eps).
full (bool, optional) – whether to compute the full loss, i.e., including the Stirling approximation term target * log(target) – target + 0.5 * log(2 * pi * target).
eps (float, optional) – default value: 1e-8
This article is sourced from: https://blog.csdn.net/shanglianlm/article/details/85019768. It is used solely for the purpose of sharing more information and does not represent the platform’s endorsement of its views or responsibility for its authenticity. Copyright belongs to the original author. If there is any infringement, please contact us for deletion.

Editor: Huang Jiyan

Proofreader: Lin Yilin

Summary of PyTorch Loss Functions

Leave a Comment