Abstract

Large language models (LLMs) can provide rich physical descriptions of most worldly objects, allowing robots to achieve more informed and capable grasping. We leverage LLMs' common sense physical reasoning and code-writing abilities to infer an object's physical characteristics—mass m, friction coefficient µ, and spring constant k—from a semantic description, and then translate those characteristics into an executable adaptive grasp policy. Using a two-finger gripper with a built-in depth camera that can control its torque by limiting motor current, we demonstrate that LLM-parameterized but first-principles grasp policies outperform both traditional adaptive grasp policies and direct LLM-as-code policies on a custom benchmark of 12 delicate and deformable items including food, produce, toys, and other everyday items, spanning two orders of magnitude in mass and required pick-up force. We then improve property estimation and grasp performance on variable size objects with model finetuning on property-based comparisons and eliciting such comparisons via chain-of-thought prompting. We also demonstrate how compliance feedback from DeliGrasp policies can aid in downstream tasks such as measuring produce ripeness.

Approach Overview

Large language models (LLMs) are able to supervise robot control in manipulation across high-level step-by-step task planning, low-level motion planning, and determining grasp positions conditioned on a given object's semantic properties. These methods inherently assume that the acts of “picking” and “placing” are straightforward tasks, and cannot acount for contact-rich manipulation tasks, such as grasping a paper airplane, deformable plastic bags containing dry noodles, or ripe produce.

In this work, we propose DeliGrasp, which leverages LLMs' . common-sense physical reasoning and code-writing abilities to infer the physical characteristics of gripper-object interactions, including mass, spring constant, and friction, to obtain grasp policies for these kinds of delicate and deformable objects. We formulate an adaptive grasp controller with slip detection derived from the inferred characteristics, endowing LLMs embodied with any current-controllable gripper with adaptive, open-world grasp skills for objects spanning a range of weight, size, fragility, and compliance.

Adaptive Grasp Policies

The minimum grasp force required to pick an object up is bounded between object slip acceleration and gripper upwards acceleration (2.5 m/s² for the UR5 robot arm), given an object's mass, m, and friction coefficient, µ.

Assuming a highly capable reasoning agent can estimate these two values, we can define a closed-loop controller to achieve a minimally-deforming grasp: increasing gripper output force F_out and decreasing gripper aperture x until sensing a contact force F_c greater than the target F_min. To determine the gain terms, ∆F_out and ∆x, i.e. how fast we close the gripper and ratchet up force, the controller utilizes an agent-determined k and ∆x to compute ∆F_out = c · k∆x (manual dampening constant c = 0.1).

We task an LLM (GPT-4) with predicting these quantities for an arbitrary object. To generate grasp policies, we leverage a dual-prompt structure similar to that of Language to Rewards, with an initial grasp “descriptor” prompt which estimates object characteristics and special accommodations, if needed, from the input object description. The “descriptor” prompt produces a structured description, which the subsequent “coder” prompt translates into an executable Python grasp policy that modulates gripper compliance, force, and aperture according to the controller described above.

DeliGrasp failures are primarily slip failures, as the simple controller is not robust to non-linearly and/or highly compliant objects such as the bag of noodles, bag of rice, spray bottle, and stuffed animal. Deforming failures did not occur, despite occasional mass overestimations, because the applied force goalpoint, F_min, is set to the estimated object's slip force rather than its maximum force. We show some of these failures below.

We also show how the compliance feedback from DeliGrasp policies enables downstream tasks such as measuring produce ripeness. In the below sequence, we show how DeliGrasp performs semantic segmentation on a bin of avocados, generates different grasp policies for different actions "checking for ripeness" vs. "picking up", and uses the measured compliance to pick an appropriate avocado for "guacamole today".

FAQs

How does this differ from Language to Rewards and Code as Policies?

DeliGrasp's primary function is the estimation of object mass, friction, and compliance, and modulating the computed grasp force depending on the high-level task description. We pair it with an adaptive grasping algorithm derived from traditional adaptive grasp controllers and tailored for these estimated characteristics. DeliGrasp parameterizes low-level contact rich manipulation, rather than directly design low-level motion with reward functions or pre-defined robot primitive skills.

How does the controller compare to a classic adaptive grasping methods?

1. DeliGrasp outperforms classical methods on the evaluated delicate objects.
2. Classical methods are often customized for specific gripper hardware or suites of sensors, whereas ours can be adapted to any current-controllable end-effector.
3. Our algorithm is a simple closed-loop feedforward controller because it relies on LLM-estimated terms which reduce controller complexity.
4. Many methods employ in-motion slip detection, which DeliGrasp is not capable of.

What else can DeliGrasp pick up?

But can DeliGrasp pick up deli meats?

Prompts

DeliGrasp: Grasp Descriptor | Grasp Descriptor with CoT Prompting | Grasp Coder

BibTeX Citation

Acknowledgements

The authors would like to thank Eric Xue, Enora Rice, Shivendra Agrawal, James Watson, and Max Conway for their feedback and support.

The website template is adapted from Language to Rewards and ProgPrompt.