Advanced PyTorch: Tensors and Gradients

Excerpt from medium

Author: Aakash N S

Machine Heart Compilation

PyTorch is an open-source neural network library developed and maintained by Facebook. It has gained significant momentum recently, with more and more developers writing tutorials for it, and this article is one of them. This is the first article in the ‘PyTorch: Zero to GANs’ tutorial series, which introduces the basic components of PyTorch models: tensors and gradients.

The complete series includes:

1. PyTorch Basics: Tensors & Gradients (this article)

2. Linear Regression & Gradient Descent: https://medium.com/jovian-io/linear-regression-with-pytorch-3dde91d60b50

3. Classification with Logistic Regression: https://medium.com/jovian-io/image-classification-using-logistic-regression-in-pytorch-ebb96cc9eb79

4. To be continued… (Neural Networks, CNNs, RNNs, GANs, etc.)

This series aims to help users better utilize PyTorch to learn about deep learning and neural networks. This article will introduce the basic components of PyTorch models: tensors and gradients.

System Setup

This tutorial adopts a code-first approach to learning PyTorch, and you should try to run and experiment with the code yourself. We will use the Anaconda distribution for Python to install the libraries and manage the virtual environment. For interactive coding and experiments, we will use Jupyter Notebook. All Jupyter Notebooks in this series can be obtained from Jovian (a collaborative platform for Jupyter). The notebook for this article can be found at: https://jvn.io/aakashns/e5cfe043873f4f3c9287507016747ae5

By running individual commands directly inside Jupyter, Jovian makes it easy to share Jupyter Notebooks in the cloud. It also captures the Python environment and libraries you need to run the notebook, allowing anyone (including yourself) to reproduce your work.

The steps are as follows:

1. Install Anaconda according to the following guide. You may also need to add the Anaconda binaries to your PATH system to run the conda command line tool.

Guide link: https://conda.io/projects/conda/en/latest/user-guide/install/index.html

2. Install the jovian Python library by running the following command (do not include the $) in a Mac/Linux terminal or Windows command prompt:

$ pip install jovian --upgrade

3. Download the notebook for this article using the jovian clone command:

$ jovian clone e5cfe043873f4f3c9287507016747ae5

This creates a directory 01-pytorch-basics, which contains the Jupyter notebook and the Anaconda environment folder.

$ ls 01-pytorch-basics
01-pytorch-basics.ipynb  environment.yml

4. Now we can open the directory and install the required Python libraries (Jupyter, PyTorch, etc.) with a single conda command:

$ cd 01-pytorch-basics
$ conda env update

5. Activate the virtual environment by running the following command:

$ conda activate 01-pytorch-basics

For older versions of conda, you may need to run the command: source activate 01-pytorch-basics.

6. Once the virtual environment is activated, we can start Jupyter by running the following command:

$ jupyter notebook

7. Now, you can access the Jupyter web interface by clicking on the link displayed in the terminal or visiting http://localhost:8888.

Then, you can click on the 01-pytorch-basics.ipynb folder to open it and run the code. If you want to input the code yourself, you can also create a new notebook by clicking the ‘New’ button.

First, import PyTorch:

Tensors

Essentially, PyTorch is a library for handling tensors. A tensor is a number, vector, matrix, or any n-dimensional array. We create a tensor with a single number:

4. is a shorthand for 4.0. It indicates that you want to create a floating-point number in Python (and PyTorch). We can verify this by checking the dtype attribute of the tensor:

We can try to create a more complex tensor:

Tensors can have any number of dimensions, with each dimension having different lengths. We can use the .shape attribute of the tensor to see the length of each dimension.

Tensor Operations and Gradients

We can combine tensors with common arithmetic operations, as follows:

We have created 3 tensors: x, w, and b. The parameters w and b have an additional parameter requires_grad set to True. We will see what it can do shortly.

By combining these tensors, we can create a new tensor y.

As expected, y is a tensor with a value of 3 * 4 + 5 = 17. The special thing about PyTorch is that we can automatically compute the derivative of y with respect to the tensors (where requires_grad is set to True), namely w and b. To compute the derivative, we can call the .backward method on the result y.

The derivative of y with respect to the input tensors is stored in the .grad attribute of the corresponding tensor.

As expected, the value of dy/dw is the same as x (i.e., 3), and the value of dy/db is 1. Note that the value of x.grad is None because x does not have requires_grad set to True. The ‘grad’ in w_grad represents the gradient, which is another term for derivative, mainly used in matrix operations.

Interoperability with Numpy

Numpy is a popular open-source library for mathematics and scientific computing in Python. It supports efficient operations on large multi-dimensional arrays and has a large ecosystem of libraries that support various tasks. These libraries include:

• Matplotlib for plotting and visualization

• OpenCV for image and video processing

• Pandas for file I/O and data analysis

PyTorch does not reinvent the wheel but interacts well with Numpy to leverage its existing tools and library ecosystem.

You can convert Numpy arrays to PyTorch tensors using torch.from_numpy.

You can then verify that the Numpy array and PyTorch tensor have similar data types.

You can use the tensor’s .to_numpy method to convert a PyTorch tensor back to a Numpy array.

The interoperability between PyTorch and Numpy is crucial because most datasets you will use are likely to be read and preprocessed as Numpy arrays.

Submitting and Uploading Notebooks

The final step is to save and submit your work using the jovian library

Jovian uploads the notebook to https://jvn.io (https://jvn.io/), capturing the Python environment for your notebook and creating a shareable link. You can use this link to share your work, allowing anyone to easily reproduce it using the jovian clone command. Jovian also has a powerful commenting interface for discussions and comments on specific parts of your notebook:

Further Reading

PyTorch tensors support many operations, and the list provided here is not exhaustive. If you want to learn more about tensors and tensor operations, you can refer to the following link:

Link: https://pytorch.org/docs/stable/tensors.html

If you want to leverage the advantages of an interactive Jupyter environment to experiment with tensors and try various combinations of operations mentioned above, consider the following:

1. What if one or more of ‘x’, ‘w’, or ‘b’ are matrices instead of numbers in the examples mentioned above? How would the resulting ‘y’, gradients w.grad, and b.grad look?

2. What if ‘y’ is a matrix created with torch.tensor, where each element represents a combination of the number tensors ‘x’, ‘w’, and ‘b’?

3. What if we have a chain of operations, not just one, such as y = x * w + b, z = l * y + m, e = c * z + d? What happens when we call e.backward()?

If you are interested in this and want to learn more about matrix derivatives, you can refer to:

Link: https://en.wikipedia.org/wiki/Matrix_calculus#Derivatives_with_matrices

Thus, we have completed the discussion on tensors and gradients in PyTorch, and the next topic will be linear regression.

This series of articles is primarily inspired by the following two articles:

• Yunjey Choi, PyTorch Tutorial for Deep Learning Researchers

• Link: https://github.com/yunjey/pytorch-tutorial

• Jeremy Howard, FastAI development notebooks

• Link: https://github.com/fastai/fastai_docs/tree/master/dev_nb

This article is compiled by Machine Heart, please contact this public account for authorization to reprint。

✄————————————————

Join Machine Heart (Full-time Reporter / Intern): [email protected]

Submissions or Reporting Requests: content@jiqizhixin.com

Advertising & Business Cooperation: [email protected]