Multimodal Models

Open Source Breakthrough! In-Depth Analysis of DeepSeek Janus Pro Multimodal Model

by
Open Source Breakthrough! In-Depth Analysis of DeepSeek Janus Pro Multimodal Model

As the Spring Festival approaches, DeepSeek is once again making strides by launching the multimodal open-source large model—Janus Pro. Although its performance is not as stunning as the previous R1, it still attracts attention due to its comprehensive architectural optimizations and open-source features. This article provides an in-depth interpretation of the technical innovations and experimental … Read more

The Evolution of DeepSeek’s Janus Series Multimodal Models

by
The Evolution of DeepSeek's Janus Series Multimodal Models

Introduction From many people’s perspective, DeepSeek’s intensive release of multimodal open-source models before the Spring Festival aims to capitalize on the momentum to take away “ClosedAI”. However, when I checked GitHub, I found that the previous Janus Flow was already several months old, and this Pro version is merely an “ordinary” upgrade for them. It … Read more

DeepSeek Janus-Pro: Breakthroughs and Innovations in Multimodal AI Models

by
DeepSeek Janus-Pro: Breakthroughs and Innovations in Multimodal AI Models

Click the “Blue Word” to Follow Us In recent years, significant progress has been made in the field of artificial intelligence, especially in the area of multimodal models. Multimodal models can process and understand various types of data, such as text and images, simultaneously, greatly expanding the application scenarios of AI. The latest model released … Read more

Complete Interpretation: From DeepSeek Janus to Janus-Pro!

by
Complete Interpretation: From DeepSeek Janus to Janus-Pro!

Datawhale Insights Author: Eternity, Datawhale Member Take Home Message: Janus is a simple, unified, and scalable multimodal understanding and generation model that decouples visual encoding from multimodal understanding and generation, alleviating potential conflicts between the two tasks. In the future, it can be expanded to incorporate more input modalities. Janus-Pro builds on this foundation, optimizing … Read more

Agentic AI: Key Technologies in Artificial Intelligence

by
Agentic AI: Key Technologies in Artificial Intelligence

In the wave of rapid development in artificial intelligence, we have witnessed the rise of generative AI and the widespread adoption of large-scale multimodal models (LMM). However, with the advancement of technology and the expansion of application scenarios, Andrew Ng proposed Agentic AI in his speech at BULIT 2024, which is becoming a new focus … Read more

Overview of Multimodal Large Models

by
Overview of Multimodal Large Models

Previously, we introduced the Large Language Models (LLMs) technology principles and applications. LLMs are a type of Foundation model, and besides LLMs, Foundation models also include Large Vision Models and Large Multimodal Models. Currently popular text-to-image models like Stable Diffusion, DALL-E, text-to-video model Sora, image-text retrieval, and visual content generation all fall under the category … Read more

New Research: MoE + General Experts Solve Conflicts in Multimodal Models

by
New Research: MoE + General Experts Solve Conflicts in Multimodal Models

Hong Kong University of Science and Technology & Southern University of Science and Technology & Huawei Noah’s Ark Lab | WeChat Official Account QbitAI Fine-tuning can make general large models more adaptable to specific industry applications. However, researchers have now found that: Performing “multi-task instruction fine-tuning” on multimodal large models may lead to “learning more … Read more

Hong Kong Tests Medical Multimodal Large Model

by
Hong Kong Tests Medical Multimodal Large Model

Better integration and exploration of industry data is expected to provide new possibilities for the development of multimodal large models in vertical fields. How can the traditional research advantages of Hong Kong connect with industrial opportunities, and how can international channels collaborate with local resources? By｜《财经》special correspondent in Hong Kong, Jiao Jian Editor｜Su Qi The … Read more

Hugging Face’s Experiments on Effective Tricks for Multimodal Large Models

by
Hugging Face's Experiments on Effective Tricks for Multimodal Large Models

MLNLP community is a well-known machine learning and natural language processing community at home and abroad, covering domestic and foreign NLP master’s and doctoral students, university teachers, and corporate researchers. The community’s vision is to promote communication and progress between the academic and industrial circles of natural language processing and machine learning at home and … Read more

HuggingFace’s Experiments on Effective Tricks for Multimodal Models

by
HuggingFace's Experiments on Effective Tricks for Multimodal Models

Xi Xiaoyao Technology Says Original Author | Xie Nian Nian When constructing multimodal large models, there are many effective tricks, such as using cross-attention mechanisms to integrate image information into language models or directly combining image hidden state sequences with text embedding sequences as inputs to the language model. However, the reasons why these tricks … Read more