Click the “Blue Word” to Follow Us
In recent years, significant progress has been made in the field of artificial intelligence, especially in the area of multimodal models. Multimodal models can process and understand various types of data, such as text and images, simultaneously, greatly expanding the application scenarios of AI. The latest model released by DeepSeek (DeepSeek-V3 Deep Analysis: A Comprehensive Interpretation of the Next Generation AI Model), Janus-Pro, represents a major breakthrough in this field. This article will delve into the technical features, innovations, and performance of Janus-Pro in multimodal tasks.
The Background of Janus-Pro’s Birth
In the AI field, the development of multimodal models has faced numerous challenges. Traditional multimodal models typically use the same visual encoder to handle both image understanding and image generation tasks. However, these two tasks require entirely different processing approaches: image understanding requires the model to extract semantic information from images, while image generation requires the model to generate high-quality images based on text descriptions. Using a single encoder to handle both tasks often leads to performance compromises.
The Janus-Pro model from DeepSeek was born to address this issue. Janus-Pro decouples the visual encoding process, handling image understanding and image generation tasks separately, thus avoiding the performance bottleneck caused by a single encoder. This innovation not only enhances the overall performance of the model but also provides new ideas for the future development of multimodal models.
The Core Architecture of Janus-Pro
The core architecture of Janus-Pro can be simply summarized as “decoupled visual encoding with unified Transformer.” Specifically, Janus-Pro employs a dual-encoder architecture, used separately for image understanding and image generation tasks, and seamlessly integrates both through a shared autoregressive Transformer.
2.1 Image Understanding Encoder
In the image understanding task, Janus-Pro utilizes the SigLIP encoder to extract high-dimensional semantic features from images. The SigLIP encoder can transform images from a two-dimensional pixel grid into a one-dimensional sequence, similar to “translating” the information in the image into a format that the model can understand. Subsequently, these features are mapped to the input space of the language model through an Understanding Adaptor, enabling the model to combine image information with text information for processing.
This process is akin to converting landmarks such as roads and buildings on a map into coordinates that a GPS system can understand. The role of the understanding adapter is to convert image features into a “language” that AI can process, achieving deep understanding of the images.
2.2 Image Generation Encoder
In the image generation task, Janus-Pro employs a VQ (Vector Quantization) encoder to convert images into discrete ID sequences. These ID sequences are mapped to the input space of the language model through a Generation Adaptor, and the model generates new images using an internal prediction head. The VQ encoder’s function is similar to converting a song into sheet music, with the model regenerating images based on these “scores”.
By assigning image understanding and image generation tasks to different encoders, Janus-Pro avoids conflicts that arise when a single encoder processes both tasks, thereby enhancing the model’s accuracy and the quality of image generation.
Optimization of Janus-Pro’s Training Strategy
In addition to architectural innovations, Janus-Pro has also undergone significant optimizations in its training strategy. The DeepSeek team (DeepSeek-R1 Distillation Models and How to Run DeepSeek-R1 Locally with Ollama) has gradually enhanced the model’s multimodal understanding and image generation capabilities through a three-phase step-by-step training process.
3.1 Phase One: Training of Adapters and Image Heads
In the first phase, Janus-Pro primarily trains the adapters and image prediction heads, focusing on the ImageNet dataset. By increasing the number of training steps, the model can better understand the dependencies between pixels, leading to more reasonable image generation. This phase of training is akin to an athlete’s foundational strength training, laying a solid groundwork for subsequent complex tasks.
3.2 Phase Two: Unified Pretraining
In the second phase, Janus-Pro abandons the ImageNet dataset and instead uses a richer text-to-image dataset for unified pretraining. This phase of training is more efficient, allowing the model to learn directly from detailed text descriptions how to generate images. This training method is similar to letting a chef directly start cooking complex dishes instead of just practicing basic ingredient combinations.
3.3 Phase Three: Supervised Fine-tuning
In the third phase, Janus-Pro further optimizes its multimodal understanding and image generation capabilities by adjusting the data ratios. By reducing the proportion of text-to-image data, the model enhances its multimodal understanding while maintaining high-quality image generation. This adjustment is akin to a student reasonably allocating study time across different subjects to achieve comprehensive development.
Data Expansion and Model Scaling
To further enhance the model’s performance, the DeepSeek team (Developing RAG Systems Based on DeepSeek R1 and Ollama (Including Code)) has also conducted extensive work in data expansion and model scaling.
4.1 Expansion of Multimodal Understanding Data
Janus-Pro has added approximately 90 million multimodal understanding data points, covering various types of data such as image descriptions, tables, charts, and documents. The inclusion of this data enables the model to better understand complex image content and extract useful information. For example, by learning from image description data, the model can better understand the scenes and objects within images; by learning from table and chart data, the model can better handle structured information.
4.2 Optimization of Image Generation Data
In terms of image generation, Janus-Pro has added about 72 million synthetic aesthetic data points, achieving a 1:1 ratio of real to synthetic data. The inclusion of synthetic data not only enhances the stability of image generation but also significantly improves the aesthetic quality of generated images. By using high-quality synthetic data, the model can converge more quickly and generate more stable and visually appealing images.
4.3 Model Scale Expansion
Janus-Pro offers models with parameter scales of 1B and 7B, with the 7B model demonstrating particularly outstanding convergence speed and performance. By increasing the model parameters, Janus-Pro can learn patterns in the data more quickly and handle more complex tasks. This expansion proves that Janus-Pro’s decoupled encoding method is also effective in large-scale models.
Performance of Janus-Pro
Janus-Pro’s performance in multimodal understanding and image generation tasks is impressive. According to the performance report released by DeepSeek (Deep Analysis of DeepSeek R1: The Synergistic Power of Reinforcement Learning and Knowledge Distillation), Janus-Pro has achieved leading results in multiple benchmark tests.
5.1 Multimodal Understanding Tasks
In the GenEval benchmark test, Janus-Pro-7B achieved an accuracy of 84.2%, surpassing competitors such as DALL-E 3 and SDXL. This result indicates that Janus-Pro has a significant advantage in understanding complex text descriptions and generating high-quality images.
5.2 Image Generation Tasks
In the DPG-Bench benchmark test, Janus-Pro-7B achieved an accuracy of 84.1%, far exceeding models like DALL-E 3 and Emu3-Gen. This achievement demonstrates Janus-Pro’s powerful capabilities in handling complex text-to-image generation tasks.
Limitations of Janus-Pro
Despite its outstanding performance in multimodal tasks, Janus-Pro still has some limitations. Firstly, the resolution of the input and output images is limited to 384×384 pixels, which somewhat affects the detail representation of images, especially in tasks requiring high-resolution outputs (such as optical character recognition). Secondly, Janus-Pro still struggles to generate realistic human images, which limits its performance in applications requiring highly realistic human depictions.
The release of DeepSeek Janus-Pro marks a new era for multimodal AI models. By decoupling visual encoding, optimizing training strategies, and expanding data and model scales, Janus-Pro has made significant advancements in multimodal understanding and image generation tasks. Although it still has some limitations, its innovative architecture and efficient training strategies provide valuable experience for the future development of multimodal models. Overall, the success of Janus-Pro proves that breakthroughs in the AI field do not always rely on disruptive innovations; sometimes, optimizing existing architectures and training methods can also yield remarkable results.
git:https://github.com/deepseek-ai/Janus
