Data & AI entrepreneur, investor Oliver Molander recently joked in a LinkedIn post: “If you had asked AI experts what LLMs were before the launch of ChatGPT in 2022, many might have answered that it’s a law degree.” He added that many people find it hard to accept that AI is much more than just LLMs and text-to-video models.
When it comes to processing tabular data and interpreting information, the real winner is XGBoost (also known as Extreme Gradient Boosting). Amid the hype surrounding numerous deep learning techniques, including large language models (LLMs) and the recently emerging retrieval-augmented generation (RAG) technologies, XGBoost excels in every aspect. The release of XGBoost 2.0 last October showed even better performance on several new classification tasks.
Although technologies like XGBoost, deep learning, and RAG cannot be directly compared, their functions are similar—they are all aimed at retrieving, understanding information, and generating outputs.
Have You Heard About the New XGBoost LLM?
Despite the tremendous advancements in generative AI and large language models (LLMs), the practical utility of XGBoost in areas relying on tabular data remains unmatched. The interpretability, efficiency, and robustness of XGBoost make it an indispensable tool across various applications, from finance to healthcare.
However, the hype surrounding LLMs and RAG (retrieval-augmented generation) technology has led to a neglect of the importance of other ML technologies, such as XGBoost. Venture capitalists are eager to ride the GenAI and LLM wave, often mislabeling every new term as a new type of LLM.
In reality, a significant portion of the return on investment is concentrated in predictive ML technologies like XGBoost and random forests. Currently, most commercial AI/ML use cases are completed using proprietary tabular business data.
When dealing with tabular datasets, efficiency is crucial. The versatility of XGBoost extends beyond classification tasks to include regression and ranking tasks. Whether you need to predict a continuous target variable, rank items based on relevance, or classify data into multiple categories, XGBoost handles it effortlessly.
The interpretability, efficiency, and versatility of XGBoost make it the preferred choice for many predictive modeling attempts, especially those relying on tabular data. In contrast, the evolving capabilities of LLMs and the enhancement potential of RAG offer enticing prospects for knowledge-intensive applications.
RAG Is Great, But the Problem Is—
A study conducted in July 2022 analyzed 45 medium-sized datasets, revealing that tree-based models like XGBoost and random forests continue to outperform deep neural networks when applied to tabular datasets.
This study was like a technical competition, where tree-based models reaffirmed their dominance in the realm of tabular data.
The emergence of RAG technology was in 2020 when a brilliant team at Meta AI decided to add a splash of color to the world of large language models (LLMs).
RAG is like a new star, its appearance changed the game. The design intention of RAG is to empower LLMs with urgently needed information retrieval techniques to address the troubling hallucination issues. In short, RAG not only breathed new life into LLMs but also brought new hope and possibilities to the entire AI field.
RAG technology offers an innovative way of processing data for large language models (LLMs), allowing users to introduce new datasets to provide the model with up-to-date information to generate answers. This technology is sometimes referred to as “advanced prompt engineering.” It is exactly what businesses need to generate insights from their own data. However, this technology has not completely resolved the hallucination issues in LLMs. On the contrary, as people begin to trust these models more, this problem may become more pronounced.
While RAG technology offers tremendous potential, its deployment is not without challenges, especially those related to data privacy and security. For instance, the presence of prompt injection vulnerabilities emphasizes the need for robust security measures when utilizing RAG-supported models. These challenges require developers and businesses to adopt more detailed and meticulous measures when implementing RAG technology to ensure user data privacy and security while complying with relevant laws and regulations.
The Territories of Large Models and XGBoost
In the machine learning (ML) ecosystem, there have traditionally been two distinct groups: one focuses on tabular data scientists using tools like XGBoost and LightGBM; the other consists of researchers working on large language models (LLMs). These two groups utilize different technologies and models. Damein Benveniste stated on LinkedIn’s The AiEdge: “I have always been a huge fan of XGBoost! For a time, I was more of an XGBoost modeler than just a machine learning modeler.”
Large language models (LLMs) generate text outputs, but the emphasis here is on utilizing the internal embeddings generated by LLMs (latent structure embeddings) that can be fed into traditional tabular models like XGBoost. While Transformers have undoubtedly revolutionized generative AI, their strength lies in handling unstructured data, sequential data, and tasks involving complex patterns.
Krishna Rastogi, CTO of MachineHack, stated: “Transformers are like the hydrogen bomb in the field of machine learning, while XGBoost is the reliable sniper rifle. When it comes to tabular data, XGBoost has proven to be the preferred precise shooter.”
Wondering if the Demo is just a demonstration? Testing the world’s first AI engineer Devin: There are still many drawbacks; it won’t take away programmers’ jobs! Zhou Hongyi temporarily wins!
The father of the World Wide Web speaks rarely: A certain giant company will be split; humans no longer need to go online, the next generation of the internet will be run by AI, and data will no longer be controlled by platforms.
