BUMBLE
We are moving towards a world where we will see many robots capable of performing complex multi-step tasks at home and in other environments, but so far, we haven’t seen many attempts to truly accomplish this in open vocabulary tasks.
Now, we have BUMBLE, which has over 90 hours of evaluation and user research! This is, in many ways, the most compelling evidence I’ve seen that we can indeed build powerful home assistants using existing technology.
What do I mean by “capable”? I mean that robots can accept open-ended tasks like “pick up a bottle of sugar-free soda” and actually complete the task, as shown in the video below. This involves all our favorite parts of the robot stack: localization, mapping, grasping, and of course, language and image understanding.
Paper link:
https://arxiv.org/abs/2410.06237
Code:
https://github.com/UT-Austin-RobIn/BUMBLE
Lucky Robots
Lucky Robots: Robots can also have their own “Spa day”! 🤖💆♂️
Here, robots are not just being trained; they are enjoying a luxurious “robot training camp” experience! We use cutting-edge technology like Unreal Engine 5.3, Open3D, and YOLOv8 to create a virtual five-star resort for them, allowing each robot to rejuvenate and be in top shape to face the challenges of the real world.
Our training framework is not a cold mechanical practice; it’s more like a “robot paradise”, ensuring zero mistakes and zero harm. The so-called “gentle” training makes us the “spiritual mentors” of the robot world! There will be no scenes of humans wielding metal rods because in “Lucky Robots”, every robot feels unprecedented care. When they leave the training camp, they feel as lucky as if they won the lottery—no injuries, full of happiness! 🎰🤣
Jokes aside, if you are an AI/ML developer or want to design the next “vacuum robot”, or are just interested in the field of robotics, you don’t need to spend thousands of dollars on a physical robot or turn your living room into a “laboratory”. With Lucky Robots, you can learn how to train robots, simulate their behaviors, achieving up to 90% accuracy. Finally, you can directly contact robot manufacturers to bring your ideas to life and start your entrepreneurial journey.
Rest assured, our luxurious training process does not harm the robots’ bodies or “souls”. That’s why we call it “Lucky Robots”!
Code:
https://github.com/luckyrobots/luckyrobots
AnyDressing
A multi-clothing virtual try-on project by ByteDance and Tsinghua: AnyDressing, which supports trying on multiple pieces of clothing simultaneously and can handle complex clothing combinations, maintaining good detail and fit.
1. For example, trying on a top + pants + coat can be completed at once, with strong customization, capable of handling various clothing combinations and personalized text prompts.
2. Suitable for various scenarios, supporting both realistic style generation and anime style.
3. Can be used in conjunction with other AI tools (ControlNet, LoRA, etc.), supporting text descriptions to adjust generation effects, such as changing the style of clothes, character expressions, and so on.
Project:
https://crayon-shinchan.github.io/AnyDressing/
Code:
https://github.com/Crayon-Shinchan/AnyDressing
Roo-Cline
The open-source alternative to Cursor has arrived quickly! Roo-Cline not only has the original features of Cursor but also supports command-line interaction and can open a browser for AI-based interactive testing!
For specifics, you can check around the 50-second mark in the video, when the AI requests to open the browser for testing after programming.
Code:
https://github.com/RooVetGit/Roo-Cline
LatentSync
LatentSync: A new benchmark for lip-sync in the open-source domain
ByteDance’s powerful open-source LatentSync may be the most effective lip-sync model available today. Previously, the industry mainly relied on Wav2Lip, and many virtual human projects were based on Wav2Lip for secondary training. However, the arrival of LatentSync has completely changed this landscape.
LatentSync is built on Stable Diffusion and has been deeply optimized for temporal consistency, significantly enhancing the matching effect between lip movements and speech. Moreover, it only requires 6.5G of VRAM for inference, making it easy to run even on resource-constrained devices, providing broader application support.
Whether for virtual humans, digital anchors, or other voice-driven animation scenarios, LatentSync can help developers create more realistic and smooth visual experiences with superior performance and effects.
Code:
https://github.com/bytedance/LatentSync