Warming-up and Shaping Robot Policy Learning with Large Language Model-Based Critic Feedback

Abstract

Reinforcement Learning (RL) enables robots to learn to perform tasks, but suffers from poor sample efficiency, especially in complex and high-dimensional environments. Due to the rich commonsense knowledge and advanced reasoning capabilities, recent work has explored the integration of robot learning via RL with large language models (LLMs). In this paper, we present Value Initialization and Adaptive Shaping (VIAS), a framework that uses large language models as external critics to provide guidance for value estimation. VIAS is expected to enhance the sample efficiency of robot policy learning via RL by warming-up and shaping its value estimation. We evaluated VIAS in a VirtualHome environment and on two real-world robot platforms in both motion planning and task planning tasks. Experimental results show that VIAS outperforms state-of-the-art methods in both learning speed and task completion. In addition, further analysis shows that VIAS remains effective regardless of the specific LLM model employed and outperforms the LLM directly gener- ated policy, demonstrating its potential for real-world robot applications.

Video 1: Demonstration of Real Robot Experiments with VIAS

Video 1. Demonstration of VIAS performance in robot experiments.

This video demonstrates the learning process and task completion results of a robot completing the goal: "Goal: put plate on kitchentable and then put cup on kitchentable" with the VIAS framework. To validate the proposed approach in a real-world scenario, we conducted a real-robot experiment using a mobile manipulator tasked with setting up a table according to a specified goal. The mobile manipulator used in this demonstration consists of a Segway base for navigation, a UR5e robotic arm equipped with a Hand-E gripper mounted on the Segway base for manipulation, and an overhead RGB-D camera fixed relative to the robot for perception. This setup provides the robot with the capabilities to perceive its environment, navigate within it, and interact with objects effectively.

ALGORITHM 1. VIAS Algorithm

Algorithm 1. Full procedure of VIAS, consisting of an initialization stage where a Q-network is pre-trained using LLM-generated heuristic values, followed by a standard deep Q-learning phase enhanced with ongoing LLM-based shaping.

Algorithm Explanation

Algorithm 1 outlines the full procedure for VIAS, consisting of an initialization stage where a Q-network is pre-trained using LLM-generated heuristic values, followed by a standard deep Q-learning phase enhanced with ongoing LLM-based shaping.

Initialization Phase (Lines 1–7) We begin by setting up an empty initial replay buffer D₀. This buffer is used to store observation-action pairs along with their corresponding heuristic values obtained from the LLM before training. For each observation-action pair (o, a) collected before the training starts, the algorithm queries the LLM to obtain a heuristic value f. This heuristic value represents the LLM's estimated cumulative reward for taking action a in observation o. The tuple (o, a, f) is then stored in the initial replay buffer D₀. Subsequently, the algorithm initializes the Q-value function Q(o, a; θ) by minimizing the pre-training loss function L₀(θ).

Training Phase (Lines 9–23) We train the agent over T timesteps. As in the initialization phase, VIAS initializes a replay buffer D. At each time step within an episode, the agent selects an action a based on the current Q-values Q(o, a). This selection is typically performed using a policy derived from Q, such as the ϵ-greedy policy. In our experiment, we used a decayed ϵ-greedy policy over timesteps. The selected action a is then executed in the environment, resulting in the observation of an immediate reward r and the next observation o′. Querying the LLM to obtain a heuristic value f for the current observation-action pair, the agent stores the experience in the replay buffer D. A mini-batch of experiences is then sampled from the replay buffer D for training, using prioritized experience replay. The Q-value Q(o, a) is then updated by incorporating both the TD error and the heuristic adjustment from the LLM. When the agent achieves the goal, the environment is initialized by setting the initial observation o.

TABLE 1. Key Hyperparameters Used in VIAS

Table 1. Key parameters used in VIAS.

Table 1 summarizes the key hyperparameters used in our VIAS implementation based on the QVIAS agent. We highlight components that are particularly relevant to the integration of Large Language Model (LLM) feedback and TDQN.

FIGURE 1. Q-value Comparison

Figure 1. Three-Dimensional Analysis of Action Selection Policies Across Three Sequential States.

This figure presents a comparative analysis of Q-values generated by different frameworks for various actions under the same state in the VirtualHome environment. The agent's Goal is: "Goal: put 1 cutleryfork on the kitchentable." The Consecutive State axis represents three sequential states experienced by the agent under the optimal policy: "Agent is in Bathroom," "Agent is in Kitchen," and "Agent is close to Cutleryfork." The Valid Actions axis enumerates all executable actions available to the agent across these states. The Q-value axis displays the normalized Q-values scaled to the interval [0,1]. We compare the Baseline TDQN—one is in its initial training phase and another one is under the final optimized policy—with an LLM-based approach that directly generates heuristic Q-values for each state–action pair. Actions marked with a triangle on the bars indicate the ones selected by the agent in the corresponding state. The results demonstrate that the TDQN agent in the early training phase exhibits a near-random Q-value distribution and suboptimal action choices. In contrast, the LLM-driven heuristic values show strong alignment with the TDQN's final optimized policy. This indicates that the large language model can effectively comprehend the task environment and provide reasonable, informed heuristic Q-values, offering robust support for the integration of LLMs in reinforcement learning frameworks.

Q-Value Evaluation Prompt