Milvus Migration and Upgrade Guide

Since its open-source release on October 15, 2019, Milvus has evolved from version 0.5.0 to the current version 0.10.2, going through numerous updates in between. Many users may have the following questions:

• Which versions can be smoothly upgraded?

• How to perform a smooth upgrade between compatible versions?

Additionally, some users may conduct functional testing in an online environment, but due to the specific nature of their deployment and data confidentiality, they can only deploy in an offline environment. This leads to the question:

• How to migrate Milvus from an online to an offline environment?

This article aims to address these questions one by one.

| Introduction to Milvus Data Directories

Whether upgrading versions or migrating from an online to an offline environment, the process revolves around Milvus’s four directories: conf, db, wal, logs. Therefore, it’s essential to introduce these four directories before we begin.

• conf

The conf directory stores Milvus’s configuration files.

In versions of Milvus up to v0.8.0, the conf directory contains two configuration files: server_config.yaml and log_config.conf. The server_config.yaml sets the configuration parameters for the Milvus server, while log_config.conf sets the logging configuration parameters for Milvus.

In versions after Milvus v0.8.0, the conf directory contains only one configuration file: server_config.yaml. The logging parameters have been moved to the logs section within server_config.yaml.

• db

The db directory is used to store Milvus’s data files and index files. If SQLite is used to store metadata, the db directory will also contain Milvus’s metadata. This directory is crucial during Milvus version upgrades and data migrations. An example structure of a db directory is shown below:

meta.sqlite: This file stores the Milvus metadata in an SQLite database.

meta.sqlite-shm: This is the shared memory file for the SQLite database.

meta.sqlite-wal: This is the WAL (Write-Ahead Logging) file for the SQLite database.

⚠️Note: If deploying Milvus in a production environment, MySQL must be used instead of SQLite for managing metadata. For specific replacement methods, refer to Using MySQL to Manage Metadata:

https://milvus.io/cn/docs/v0.10.2/data_manage.md

tables: This folder stores all collections and partitions in Milvus.

test_table: This folder stores all data segment files within the collection named “test_table.”

test_partition: This folder stores all data segment files within the partition named “test_partition.”

⚠️Note:

• In Milvus, collections and partitions are logically parent-child, but they are at the same level in the file structure.
• The relationship between collections and partitions can be found in the meta file.

1591254355070376000: This directory stores all data segment (segment) files for a collection.

1591254355070376000.rv: This file stores the raw vectors for the current data segment (segment).

1591254355070376000.uid: This file stores all vector IDs for the current data segment (segment).

1591254661450955000: This file stores the index file for the current data segment (segment).

bloom_filter: This is a Bloom filter used to determine whether a vector ID exists in this data segment (segment).

deleted_docs: This is used to record deleted vector IDs.

• wal

The wal directory is used to store Milvus’s Write-Ahead Logging files, which generally do not affect version upgrades and data migrations.

• logs

The logs directory is used to store Milvus’s log files, which generally do not affect version upgrades and data migrations.

| Milvus Version Upgrade Guide

Milvus version upgrades can generally be summarized in three steps: data compatibility check, server upgrade, and client upgrade.

• Data Compatibility Check

Before upgrading Milvus, the first step is to check whether the data between the two versions is compatible. If the data is not compatible, the two versions cannot be directly upgraded; data must be re-imported for the upgrade. Generally, if the metadata of the two versions of Milvus has not changed, then the data of these two versions is compatible.

If the two versions of Milvus data are compatible, during the upgrade, you can directly copy the db directory from the old version of Milvus and map it to the new version when starting Milvus.

From the open-source version 0.5.0 to the latest version 0.10.2, based on data compatibility, these versions can be divided into three phases, with each phase’s version data being mutually compatible, while data from different phases is not compatible:

Phase One	v0.5.0 ~ v0.5.3
Phase Two	v0.6.0
Phase Three	v0.7.0 ~ v0.10.2

When upgrading versions, apart from the db directory, other folders under the Milvus directory are generally not reusable, especially configuration files. Please ensure to use the configuration files corresponding to the respective version; otherwise, Milvus will not start successfully.

• Server Upgrade

Server upgrade mainly refers to upgrading the Milvus Docker by pulling the corresponding version of the Milvus Docker image provided by the official source and using the docker run command to start the Milvus server.

• Client Upgrade

Milvus currently provides SDKs in Python, Java, C++, and Go languages. After upgrading the server, the client must also be upgraded to the corresponding version. The version correspondence between client and server can be found in the Milvus Release Notes: https://milvus.io/cn/docs/v0.10.2/release_notes.md.

| Migration of Milvus from Online to Offline Environment

For scenarios where testing is done in an online environment but deployment needs to be in an offline environment, the migration can be achieved through three steps: installing the Milvus server in the offline environment, installing the Milvus client in the offline environment, and copying the Milvus directory.

• Installing Milvus Server in Offline Environment

The official recommendation is to deploy the Milvus server using Docker, so this article mainly describes how to install the Docker version of Milvus in an offline environment. The key step for offline installation is obtaining the Milvus Docker image; once you have the Docker image, you can easily start the Milvus server using docker run.

Since Docker provides the docker save and docker load commands to help us package and load Docker images, you can pull the Milvus Docker image on an online machine, compress it into a TAR package using docker save, copy the image TAR package to the offline machine using portable storage media, and finally load the image using docker load. The entire process can be described using the flowchart below:

The corresponding Docker commands are as follows:

 $ docker save milvusdb/milvus > milvus_image.tar $ docker load < milvus_image.tar

• Installing Milvus Client in Offline Environment

Milvus provides clients in various languages; the following mainly introduces how to install the Milvus Python client pymilvus in an offline environment.

The first step: Download the pymilvus package (multiple WHL files) from an online machine to a specified folder.

pip3 download -d <your path> pymilvus

The second step: Copy the downloaded pymilvus package to the offline machine for installation.

pip3 install <your path>/*.whl
Through these two steps, you can successfully install the Milvus Python client in the offline environment.

• Copying the Milvus Directory

The final step in migrating from an online to an offline environment is to copy the Milvus folder from the online machine to the offline machine. After copying, to ensure that Milvus Docker starts smoothly, you also need to manually delete the wal folder from the Milvus directory.

Once all the preparations are complete, you can start Milvus on the offline machine by using docker run to map the copied db directory, logs directory, and conf directory.

The above is a brief introduction to some strategies for Milvus migration and upgrades. We welcome more enthusiasts promoting unstructured data search to join the Milvus open-source community and witness the growth of Milvus together!

| Welcome to Join the Milvus Community

github.com/milvus-io/milvus | Source Code

milvus.io | Official Site

milvusio.slack.com | Slack Community

zhihu.com/org/zilliz-11/columns | Zhihu

zilliz.blog.csdn.net | CSDN Blog

space.bilibili.com/478166626 | Bilibili