VLLM

Performance Comparison of Mainstream Inference Frameworks on Llama 2

by
Performance Comparison of Mainstream Inference Frameworks on Llama 2

MLNLP community is a well-known machine learning and natural language processing community both domestically and internationally, covering NLP graduate students, university teachers, and corporate researchers. The Vision of the Community is to promote communication and progress between the academic and industrial circles of natural language processing and machine learning, especially for beginners. Reprinted from | … Read more

Ali Qwen 2.5-1M Open Source: 320GB for 14B Tokens

by
Ali Qwen 2.5-1M Open Source: 320GB for 14B Tokens

Recently, domestic large models such as DeepSeek, Kimi, Baichuan Intelligence, Doubao, and Jieti Xingchen have released their respective models. On the last day of the year, Alibaba Qwen couldn’t hold back anymore and also open-sourced the million-token contextQwen2.5-1M model and its corresponding inference framework support. Open Source Model: The Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M models, which extend … Read more

Running HuggingFace DeepSeek V2 on Single Node A800

by
Running HuggingFace DeepSeek V2 on Single Node A800

0x0. Background Trying to run the DeepSeek V2 released on HuggingFace, I encountered several issues. Here are the solutions. The open-source DeepSeek V2 repo link provided by HuggingFace is: https://huggingface.co/deepseek-ai/DeepSeek-V2 0x1. Error 1: KeyError: ‘sdpa’ This issue has also been reported by the community. https://huggingface.co/deepseek-ai/DeepSeek-V2/discussions/3 Insert image description here The solution is quite simple; just … Read more