Data Loading

Multiprocessing Parallel Processing in PyTorch

by
Multiprocessing Parallel Processing in PyTorch

Source: DeepHub IMBA This article is approximately 2000 words long and is recommended to be read in 9 minutes. Understanding and utilizing multiprocessing techniques are essential for optimizing performance in PyTorch. PyTorch is a popular deep learning framework that is very convenient when using a single GPU for computation. However, when it comes to handling … Read more

Data Loading and Ingestion in LlamaIndex

by
Data Loading and Ingestion in LlamaIndex

Data is key. Before we can operate on our data with LLM, we first need to process and load the data. This is similar to ETL (Extract, Transform, Load) in machine learning engineering. In LlamaIndex, the data ingestion pipeline typically consists of three main stages: Loading Data Transforming Data Indexing and Storing Data We will … Read more