How to Deploy Private Free Large Models Locally with Ollama

Ollama is an open-source framework designed for the convenient deployment and operation of large language models (LLMs) on local machines. Its core feature is to simplify usage and provide an efficient technical architecture, allowing developers to easily access and use powerful AI language models. Ollama supports local operation, meaning users can use models without a network connection, which offers advantages in privacy and data security.

Ollama has made significant optimizations in model inference, allowing 7b models to run smoothly even on M1 chips.

1. Installing Ollama

First, download the Ollama client from ollama.com.After running the client, it prompts to install the Ollama command line

2. Basic Usage of Ollama

After installation, you can verify successful installation by entering the ollama command in the command line.Ollama provides the following commands:

• <span>ollama list: Display the model list.</span>
• <span>ollama show: Display model information.</span>
• <span>ollama pull: Pull a model.</span>
• <span>ollama push: Push a model.</span>
• <span>ollama cp: Copy a model.</span>
• <span>ollama rm: Delete a model.</span>
• <span>ollama run: Run a model.</span>
• <span>ollama serve: Start the service.</span>
• <span>ollama --help: View the complete command list.</span>

Ollama provides a wide range of models for use; here we choose the mistral 7b model

Deploying the model: run the command line to download the model

ollama run mistral

Downloading may take some time, depending on network speed. After successful execution, you can start chatting.

You can switch models or related settings with the following commands

• /set Set session variables

  /show Show model information

  /load <model> Load a session or model

  /save <model> Save your current session

  /clear Clear session context

  /bye Exit

  /?, /help Help for a command

  /? shortcuts Help for keyboard shortcuts

In addition to command line usage, Ollama also supports REST API calls

curl http://localhost:11434/api/chat -d '{  "model": "mistral",  "messages": [    { "role": "user", "content": "why is the sky blue?" }  ]}'

At the same time, the Ollama community is very active, providing a wealth of UI libraries and various plugins

3. Calling Ollama Models with Python

For further development, such as document Q&A, Python is mainly used to call the Ollama framework to utilize AI large models.

import ollama
stream = ollama.chat (    model = 'mistral',    messages = [{'role': 'user', 'content': 'Tell me a joke'}],    stream = True,)
for chunk in stream:    print(chunk['message']['content'], end='', flush=True)

4. Conclusion

Advantages of Ollama:

• Comprehensive Functionality
  
  : Ollama simplifies the model deployment and configuration process by integrating model weights, configuration files, and necessary datasets into a single file through the concept of Modelfile.
• Lightweight Design
  
  : Ollama has low resource usage during runtime, allowing it to run efficiently in resource-limited local environments. Additionally, Ollama supports hot-loading model files, providing higher flexibility.
• User-Friendliness
  
  : Ollama offers multiple installation methods covering Win, Mac, and Linux platforms,
• Rich Large Models
  
  : A large number of open-source LLM models, including those from leading domestic companies, are available, along with detailed model file information.

References

• Ollama Official Websitehttps://ollama.com/
• https://github.com/ollama/ollama