Gradient Descent

Comprehensive Summary of Optimization Algorithms in Machine Learning

by
Comprehensive Summary of Optimization Algorithms in Machine Learning

Click on the above“Beginner Learning Vision” and choose to add Star or Pin. Important content delivered in real-time. Introduction For almost all machine learning algorithms, whether supervised learning, unsupervised learning, or reinforcement learning, it generally boils down to solving an optimization problem. Therefore, optimization methods occupy a central position in the derivation and implementation of … Read more

Multi-Task Loss in Pytorch: Add or Backward Separately?

by

MLNLP ( Machine Learning Algorithms and Natural Language Processing ) community is a well-known natural language processing community at home and abroad, covering domestic and foreign NLP master’s and doctoral students, university teachers, and corporate researchers.The vision of the community is to promote communication between the academic and industrial circles of natural language processing and … Read more

From Gradient Descent to Adam: Understanding Neural Network Optimization Algorithms

by
From Gradient Descent to Adam: Understanding Neural Network Optimization Algorithms

When adjusting the way the model updates weights and bias parameters, have you considered which optimization algorithm can yield better and faster results for the model? Should you use gradient descent, stochastic gradient descent, or the Adam method? This article introduces the main differences between different optimization algorithms and how to choose the best optimization … Read more

Essentials of Andrew Ng’s DeepLearning.ai Course: Neural Networks Basics

by
Essentials of Andrew Ng's DeepLearning.ai Course: Neural Networks Basics

The following notes summarize key points from the second week of the first part of Andrew Ng’s “Neural Networks and Deep Learning” course in the DeepLearning.ai project on Coursera. These notes do not cover all the details of the video lectures. For content omitted in these notes, please refer to Coursera or NetEase Cloud Classroom. … Read more

An Overview of Linear Regression in Machine Learning

by
An Overview of Linear Regression in Machine Learning

1. Introduction to Linear Regression 1.1 Origin of Regression Francis Galton, a British biologist, studied the relationship between the heights of parents and their children. He concluded that if parents are taller than the average height of the population, their children tend to be shorter than their parents, moving closer to the average height of … Read more

BP Neural Network Algorithm in Machine Learning

by

The BP neural network (Back Propagation) is a multi-layer network that propagates signals forward and errors backward. This network has very strong computational capabilities and can solve many nonlinear problems by analyzing the nonlinear mapping relationship between input and output variables through training samples. 1. Structure of BP Neural Network The structure of the BP … Read more

Understanding Gradient Descent in Neural Networks

by
Understanding Gradient Descent in Neural Networks

This article will coverthe essence of Gradient Descent, the principles of Gradient Descentand the algorithms of Gradient Descent in three aspects, helping you understand Gradient Descent Gradient Descent | GD. Gradient Descent 1.Essence of Gradient Descent Machine Learning’s “Three Essentials”: Select a model family, define a loss function to quantify prediction errors, and find the … Read more

The Real Culprit Behind Failed Deep Neural Network Training Is Degeneracy, Not Vanishing Gradients

by
The Real Culprit Behind Failed Deep Neural Network Training Is Degeneracy, Not Vanishing Gradients

Selected from severelytheoretical Translated by Machine Heart Contributors: Jiang Siyuan, Liu Xiaokun The author demonstrates through the example of deep linear networks that the reason for the poor performance of the final network is not vanishing gradients, but rather the degeneracy of the weight matrices, which reduces the effective degrees of freedom of the model, … Read more

In-Depth Summary of 27 Common Deep Learning Questions

by
In-Depth Summary of 27 Common Deep Learning Questions

Follow our WeChat public account “ML_NLP“ Set it as “Starred“, delivering heavy content as soon as possible! Source | https://github.com/scutan90/DeepLearning-500-questions/blob/master/ch03_%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%9F%BA%E7%A1%80/%E7%AC%AC%E4%B8%89%E7%AB%A0_%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%9F%BA%E7%A1%80.md Organizer | dilligencer https://zhuanlan.zhihu.com/p/105476748 This article is for interview experience sharing only and not for commercial use. If there is any infringement, please contact for deletion. 1. Why is it difficult to train deep neural … Read more

Comprehensive Summary of Optimization Algorithms in Machine Learning

by
Comprehensive Summary of Optimization Algorithms in Machine Learning

Source: Internet Introduction For almost all machine learning algorithms, whether supervised learning, unsupervised learning, or reinforcement learning, the ultimate goal generally boils down to solving optimization problems. Therefore, optimization methods play a central role in the derivation and implementation of machine learning algorithms. In this article, the author will provide a comprehensive summary of the … Read more