Robotic Garment Handling Using Large Language Models and Behavior Trees

The potential of Large Language Model (LLM) agents to revolutionize industrial robotics is substantial, as they enable natural language to serve as a highly flexible and intuitive medium for humans to communicate tasks to robots. This research investigates the capabilities of LLM agents in guiding industrial robots, with a particular focus on their effectiveness in generating Behavior Trees (BTs) for garment handling applications.
Our framework consists of three essential components: an LLM for code generation and common-sense reasoning, a vision-language model for open-vocabulary visual recognition, and a specialized 3D object recognition model. We present a novel four-stage prompt engineering process that customizes prompts to meet the specific requirements of BT generation. This innovative process streamlines the development of programs for complex tasks, such as safety management and parallel processing within a multi-arm robotic workspace, ensuring that robots execute actions safely and reliably.
To enhance precision in grasping and manipulation, we integrate a RealSense RGB-D camera that provides essential depth information. This is combined with foreground-background segmentation and curvature analysis to facilitate accurate handling of objects. Both real-world tests and simulated trials conducted in a Gazebo environment highlight the effectiveness of this approach, showcasing the LLM's ability to improve safe robotic operations in various industrial settings. Furthermore, user interactions with the system exceeded our initial expectations, indicating a promising direction for the integration of LLMs in enhancing industrial robotics and automation.