With the rapid development of large model technology, its application scope has widely penetrated various aspects of enterprise R&D applications, production, and management. Due to the large number of parameters in large models and their complex and diverse deployment scenarios and forms, higher requirements have been put forward for the deployment, inference, and service aspects of the models. How to efficiently compress models, optimize for autoregressive characteristics, achieve distributed deployment inference, and ensure model response speed while reducing latency; how to optimize request scheduling strategies, achieve resource elasticity and scaling, and improve throughput capacity and stability to cope with dynamically changing traffic and high concurrency scenarios; how to effectively control resources and adapt costs to maximize economic benefits are all prominent new challenges in the implementation of large models.
The China Academy of Information and Communications Technology (hereinafter referred to as “CAICT”) Artificial Intelligence Research Institute pays close attention to the technical development of large model inference platforms. Relying on the key technologies and application evaluation of artificial intelligence, the Ministry of Industry and Information Technology’s key laboratory, and the AI Infra working group of the China Artificial Intelligence Industry Development Alliance, CAICT has jointly formulated the “Technical Requirements for Large Model Inference Platforms” standard in collaboration with over seventy entities including Alibaba Cloud, Ant Group, Telecom, Mobile, Unicom, China Bond Financial Technology, SenseTime, and Jinkai New Energy.
On January 8, 2025, the “Large Model Engineering Achievement Release Conference” was held in Beijing, where Yu Wenmengke from the CAICT Artificial Intelligence Research Institute released an interpretation of the “Technical Requirements for Large Model Inference Platforms”.
1. Target Audience
This standard aims to provide technical references for developers of large model inference platforms and selection references for enterprises using them.
2. Technical Specification Content for Large Model Inference Platforms
This standard outlines the functionality, performance, and stability of large model inference platforms, focusing on deployment, inference, service, and management to achieve low latency, high throughput, scalability, high usability, and low cost goals. The standard includes 81 capability items, with 35 basic functions and 46 advanced functions, covering the following core items in each capability domain.
(1)
Model Deployment
● Basic Capabilities:
Including model format support, various compression methods, deployment resource settings, deployment environment settings, containerized deployment, distributed deployment, deployment strategy settings, pre-deployment testing, etc.;
● Optimization Techniques:
Including cache optimization strategies, automated deployment, etc.
● Metrics:
Model compression: including compression ratio, accuracy loss, inference acceleration ratio, etc.
Model loading: including loading time, etc.
(2)
Model Inference
● Basic Capabilities:
Including inference engine support, inference effect control, data backflow, pre-release testing, prompt word construction/management/optimization, etc.;
● Optimization Techniques:
Including memory optimization, computational optimization, distributed optimization, scheduling optimization, etc.
● Metrics:
Latency: including first token latency, token inter-latency, total latency, etc.
Throughput: including tokens per second/query count, etc.
(3)
Platform Services and Management
● Service Scalability:
Including open-source tool support, API support, service orchestration, etc.;
● Stability:
Including metric monitoring, load balancing, elastic scaling, etc.;
● Usability:
Including various resource management/support, template/tool management support, etc.
The AI Infra working group of the China Artificial Intelligence Industry Development Alliance will be committed to promoting high-quality development in the field of artificial intelligence through technical research, technical specification formulation, policy research, etc., to facilitate supply-demand matching and application implementation, and jointly promote the high-quality development of the artificial intelligence industry with all parties.