• https://pub.towardsai.net/want-to-learn-quantization-in-the-large-language-model-57f062d2ec17

• First, understand what quantization is and why it is needed.
• Next, delve into how to perform quantization and understand it through simple mathematical derivation.
• Finally, write some PyTorch code to quantize and dequantize LLM weight parameters.

What is Quantization and Why is it Needed?

Left: Base model size calculation (unit: GB), Right: Quantized model size calculation (unit: GB)

By this calculation, a large model with 7B FP16 deployment weights 14G, INT8 is 8G, and INT4 is halved to 4G.

• Reduce memory requirements.
• Enhance inference performance.

How Does Quantization Work? Simple Mathematical Derivation

• Wmin, Wmax: Minimum and maximum values of the original tensor (data type: FP32, 32-bit float). In most modern LLMs, the default data type for weight tensors is FP32.
• Qmin, Qmax: Minimum and maximum values of the quantized tensor (data type: INT8, 8-bit integer). We can also choose other data types like INT4, INT8, FP16, and BF16 for quantization. In our example, we will use INT8.
• Scale value (S): During quantization, the scale value reduces the original tensor values to obtain the quantized tensor. During dequantization, it magnifies the quantized tensor values to obtain the dequantized values. The scale value’s data type is the same as the original tensor, which is FP32.
• Zero point (Z): The zero point is a non-zero value in the quantized tensor range that directly maps to the value 0 in the original tensor range. The zero point’s data type is INT8 because it is within the quantized tensor range.
• Quantization: The “A” section in the figure shows the quantization process, i.e., mapping from [Wmin, Wmax] to [Qmin, Qmax].
• Dequantization: The “B” section in the figure shows the dequantization process, i.e., mapping from [Qmin, Qmax] to [Wmin, Wmax].

|Zero point and Quantized tensor out-of-range

The range of quantized tensor values is -128 to 127 (INT8, signed integer data type). If the data type of the quantized tensor value is UINT8 (unsigned integer), the range is from 0 to 255.

Symmetric Linear Quantization

Quantization and Dequantization of LLM Weight Parameters

Quantization of weight parameters in transformer architecture

Example of FP32, INT8, UINT8 data type distribution and calculation

# !pip install torch; Install the torch library if you haven't already.
# Import the torch library
import torch

original_weight = torch.randn((4,4))
print(original_weight)

Original FP32 weight tensor

def asymmetric_quantization(original_weight):
    # Define the data type you want to quantize to. In our example, it is INT8.
    quantized_data_type = torch.int8
    
    # Get Wmax and Wmin values from the original FP32 weights.
    Wmax = original_weight.max().item()
    Wmin = original_weight.min().item()
    
    # Get Qmax and Qmin values from the quantized data type.
    Qmax = torch.iinfo(quantized_data_type).max
    Qmin = torch.iinfo(quantized_data_type).min
    
    # Calculate the scale value using the scaling formula. Data type - FP32.
    # If you want to understand the derivation of the formula, please refer to the mathematical section of this article.
    S = (Wmax - Wmin)/(Qmax - Qmin)
    
    # Calculate the zero point value using the zero point formula. Data type - INT8.
    # If you want to understand the derivation of the formula, please refer to the mathematical section of this article.
    Z = Qmin - (Wmin/S)
    # Check if the Z value is out of range.
    if Z < Qmin:
        Z = Qmin
    elif Z > Qmax:
        Z = Qmax
    else:
    # The data type of the zero point should be the same as the quantized value, which is INT8.
        Z = int(round(Z))
    
    # We have original_weight, scale, and zero_point now, we can use the formula derived in the mathematical section to calculate the quantized weights.
    quantized_weight = (original_weight/S) + Z
    
    # We will also round it and use the torch clamp function to ensure the quantized weights are within the range and remain between Qmin and Qmax.
    quantized_weight = torch.clamp(torch.round(quantized_weight), Qmin, Qmax)
    
    # Finally, convert the data type to INT8.
    quantized_weight = quantized_weight.to(quantized_data_type)
    
    # Return the final quantized weights.
    return quantized_weight, S, Z

def asymmetric_dequantization(quantized_weight, scale, zero_point):
# Use the dequantization formula derived in the mathematical section.
# Also, make sure to convert the quantized weights to float type because the subtraction between two INT8 values (quantized_weight and zero_point) would yield unexpected results.
    dequantized_weight = scale * (quantized_weight.to(torch.float32) - zero_point)

return dequantized_weight

quantized_weight, scale, zero_point = asymmetric_quantization(original_weight)
print(f"quantized weight: {quantized_weight}")
print("\n")
print(f"scale: {scale}")
print("\n")
print(f"zero point: {zero_point}")

Quantized weight, scale, and zero point value

dequantized_weight = asymmetric_dequantization(quantized_weight, scale, zero_point)
print(dequantized_weight)

Dequantized weight value

quantization_error = (dequantized_weight - original_weight).square().mean()
print(quantization_error)

The quantization error is very small.

• https://github.com/tamangmilan/llm_quantization/blob/main/llm_quantization_part_1.ipynb

Scan the QR code to add the assistant’s WeChat