Source: ZHUAN ZHI
This article is a book introduction, recommended reading time: 5 minutes.
This book provides a comprehensive and concise overview of the mathematical engineering of deep learning.
This book provides a comprehensive and concise overview of the mathematical engineering of deep learning. In addition to outlining the fundamentals of deep learning, the course also includes convolutional neural networks, recurrent neural networks, transformers, generative adversarial networks, reinforcement learning, and various techniques. The focus is on the basic mathematical descriptions of deep learning models, algorithms, and methods. The report mainly addresses the agnosticism of computer code, neuroscience relationships, historical perspectives, and theoretical research. The benefit of this approach is that readers with mathematical proficiency can quickly grasp the essence of modern deep learning algorithms, models, and techniques without delving into computer code, neuroscience, or historical processes.https://deeplearningmath.org/Deep learning can be easily described through mathematical language, at a level accessible to many professionals. Readers from fields such as engineering, signal processing, statistics, physics, pure mathematics, econometrics, operations research, quantitative management, applied machine learning, or applied deep learning will quickly gain insight into the key mathematical engineering components of the field.This book consists of 10 chapters and 3 appendices. Chapters 1-4 outline the key concepts of machine learning, summarize the optimization concepts required for deep learning, and highlight the fundamental models and concepts. Chapters 5-8 discuss the core models and architectures of deep learning, including fully connected networks, convolutional networks, recurrent networks, and summarize various aspects of model tuning and application. Chapters 9-10 cover specific domains, namely generative adversarial networks and deep reinforcement learning. Appendices A-C provide mathematical support. Here is a detailed overview of the content.Chapter 1 – Introduction: In this chapter, we outline deep learning, demonstrate key applications, survey the relevant high-performance computing ecosystem, discuss high-dimensional data, and set the tone for the remainder of the book. This chapter discusses key terms including data science, machine learning, and statistical learning, placing these terms within the context of the book. Major popular datasets such as ImageNet and MNIST digits are also outlined, describing the emergence of deep learning.Chapter 2 – Principles of Machine Learning: Deep learning can be viewed as a branch of machine learning, thus this chapter provides an overview of the key concepts and examples of machine learning. General concepts of supervised learning, unsupervised learning, and iterative learning optimization are introduced to the reader. Concepts such as training sets, testing sets, and the principles of cross-validation and model selection are introduced. A key object explored in this chapter is linear models, which can also be trained via iterative optimization. This allows us to see practical applications of the basic gradient descent algorithm, which is later refined and extensively used in the sequel of this book.Chapter 3 – Simple Neural Networks: In this chapter, we focus on binary classification using logistic regression and the related Softmax regression model for multi-class problems. This introduces the principles of deep learning, such as cross-entropy loss, decision boundaries, and simple backpropagation case studies. A simple nonlinear autoencoder architecture is also introduced. Model tuning aspects are discussed, including feature engineering and hyperparameter selection.Chapter 4 – Optimization Algorithms: Training deep learning models involves optimizing learning parameters. Therefore, a solid understanding of optimization algorithms is necessary, as well as an understanding of specialized optimization techniques for deep learning models (such as the ADAM algorithm). In this chapter, we will focus on these techniques as well as more advanced second-order methods that are slowly entering practice. We also explore the details of various forms of automatic differentiation and compare them in the context of logistic regression, where both first and second-order methods can be used.Chapter 5 – Feedforward Deep Networks: This chapter is the core of the book, where we define general feedforward deep neural networks. After exploring the expressive power of deep neural networks, we delve into training details through understanding the backpropagation algorithm for gradient assessment and explore other practical aspects such as weight initialization, dropout, and batch normalization.Chapter 6 – Convolutional Neural Networks: The success of deep learning is largely attributed to the power of convolutional neural networks when applied to images and similar data formats. In this chapter, we will explore the concept of convolution and understand it in the context of deep learning models. Concepts of channel and filter design are introduced, followed by discussions on commonly used advanced architectures that have made significant impacts and are still in use today. We also explore some key tasks related to images, such as object localization.Chapter 7 – Sequence Models: Sequence models are crucial for data such as text applications in natural language processing. In this chapter, we will learn key ideas in the field of deep learning. We explore recurrent neural networks and their extensions, including long short-term memory models, gated recurrent units, end-to-end language translation autoencoders, and attention models with transformers.Chapter 8 – Industry Techniques: After learning about feedforward networks, convolutional networks, and various forms of recurrent networks, we now explore common methods for tuning and integrating these models in applications. Key issues include hyperparameter selection and techniques for optimizing them. Other issues involve model adaptation from one dataset to another through transfer learning and methods for augmenting datasets. We also discuss the applications and implementation aspects of image transformers, including descriptions of deep learning software frameworks.Chapter 9 – Generative Adversarial Networks: In this chapter, we survey and explore generative adversarial networks (GANs), which are models capable of synthesizing realistic-looking fake data. The basic construction of GANs is based on a game-theoretic setup where the generator model and discriminator model are jointly trained to yield a trained system. We discuss several GAN architectures and interesting mathematical aspects that arise when adapting loss functions.Chapter 10 – Deep Reinforcement Learning: In the final chapter, we will explore the principles of deep reinforcement learning, an adaptive control method for dynamic systems. When considering artificial intelligence systems, this is often introduced in the context of agents; however, we adopt a more classical approach and present it in the context of control theory and Markov decision processes. We first lay the groundwork for MDPs and Q-learning, then explore various advancements in approximating Q functions through deep neural networks.
