“ A couple of days ago, Microsoft open-sourced the 14B dense model phi-4, which is called the strongest 14B model available. How strong is the model? What others say doesn’t count; only trying it yourself will tell. However, when you want to get hands-on, you find that most current versions of inference engines are not adapted yet, requiring you to wait for a release. For most people, performing model inference by themselves still presents a bit of a hurdle. This article will introduce how to leverage Xinference to configure it easily through simple operations, without waiting for the official adaptation of various inference engines, allowing you to effortlessly achieve local inference.”

A friend asked me a question that was somewhat unrelated to the article’s theme, “How to infer the newly released phi-4?” Thus, this topic was brought up. Next, even if you don’t have a deep technical background, as long as you have a GPU (recommended minimum single card 16GB) or you are using a MAC (minimum 16GB), you can follow along with me step by step to easily infer the phi-4 large model that has not been adapted by the inference engine.

Performance of the Phi-4 Model

Let’s first briefly understand the performance of this model; below are the benchmarks for phi-4.

I will briefly explain the meaning of the related metrics to help everyone make their own observations.

• • <span>MMLU</span>: Benchmark test for evaluating the multi-task language understanding capability of large language models.
• • <span>MATH</span>: Challenging competitive math problems.
• • <span>GPQA</span>: Complex, graduate-level scientific questions.
• • <span>DROP</span>: Complex understanding and reasoning.
• • <span>MGSM</span>: Multilingual elementary mathematics.
• • <span>HumanEval</span>: Functional code generation.
• • <span>SimpleQA</span>: Factual answering.

From the above benchmark evaluations, it can be seen that the metrics of phi-4 are indeed impressive.

Recommended installation method for Linux

Recommended installation method for Linux

Given the characteristics of AI industrial production-level toolchains, configuring the inference engine on a Linux system will provide a better experience. Below is the fully dependent installation command:

pip install "xinference[all]" -U -i https://pypi.tuna.tsinghua.edu.cn/simple --force-reinstall

Recommended installation method for Windows

Since Windows is primarily used for personal PCs rather than industrial production, there are some compatibility issues with toolchains. Below is the more compatible installation command for Windows:

pip install "xinference[transformers]" -U -i https://pypi.tuna.tsinghua.edu.cn/simple --force-reinstall

Recommended installation method for Mac

Mac has its own inference framework mlx, and Xinference has already completed integration with mlx, so everyone can easily use mlx for model inference. My daily use device is a Mac, so I chose this method to install Xinference:

pip install "xinference[mlx,transformers]" -U -i https://pypi.tuna.tsinghua.edu.cn/simple --force-reinstall

Downloading the Model

# This is a collection that includes the original phi-4 model and the quantized model

https://www.modelscope.cn/collections/phi4-Quantization-2bc1a92926ec42

• • Windows and Linux systems can download the BF16 model.
• • For Mac systems with 16GB memory, it is recommended to download the MLX-3bit version model; if the memory is above 20GB, you can download MLX-4bit

Registering the Model

After the above two steps, if all goes well, you have installed Xinference, and the phi-4 model has been downloaded. As the phi-4 model has just been open-sourced, Xinference has not yet had time to do official configuration, which gives us a chance to practice defining the model ourselves.

Starting Xinference

Now, execute the command to launch the inference engine console.

xinference-local --host 127.0.0.1 --port 9997

Access the browser at http://127.0.0.1:9997

You can select the language as Chinese in the lower left corner. Click on the left menu bar to<span>Register Model</span>, as shown in the image below.

Basic Configuration

First, as shown in the image above, configure some basic information of the model. Then scroll down the page.

As you can see in the image below:

Follow me to configure:

• 1. Model Series, since the engine has not yet adapted, phi-4 is not in the list, here select<span>phi-3-mini-128k-instruct</span> to proceed.
• 2. Chat Template, this step is very critical as it directly determines whether the inference model’s results can be displayed correctly. You must completely paste and replace the content in the editing bar with the following text.

{% for message in messages %}{% if message['role'] == 'system' %}{{'<|im_start|>system<|im_sep|>' + message['content'] + '<|im_end|>'}}{% elif message['role'] == 'user' %}{{'<|im_start|>user<|im_sep|>' + message['content'] + '<|im_end|><|im_start|>assistant<|im_sep|>'}}{% elif message['role'] == 'assistant' %}{{message['content'] + '<|im_end|>'}}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant<|im_sep|>' }}{% else %}{{ '<|endoftext|>' }}{% endif %}

Windows: Go to the path under your account folder, usually:

C:\Users\(your account name)\.xinference\model\llm

Linux: Go to the path under your account folder, usually:

~/.xinference/model/llm

After clicking, configure for Windows and Linux systems as shown in the image below.

Configure for MAC systems as shown in the image below.

Click the rocket button below to start the model.

Wait a moment; after the model loads, it will jump to the interface below.

Click the Chat button in the red box to start chatting.

After chatting, remember to click the “trash can” button on the right to clear the model from memory.