UniPred
01
Concepts from demonstrations
Demonstrations are labelled with actions, not concepts. An agent in the loop invents the concepts that explain what each action changes.
IEEE Transactions on Robotics · 2026
UniPred
Learning the concepts behind robot tasks from demonstrations, and planning with them over long horizons.
Unifying Deep Predicate Invention with Pre-trained Foundation Models
* Equal contribution · † Corresponding author
Affiliations
1 Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA
2 Computer Science and Engineering Division, University of Michigan
3 Department of Computer Science, University of Pittsburgh
4 Centaur AI Institute
5 Department of Electrical and Computer Engineering, Princeton University
6 Department of Civil and Environmental Engineering, University of Michigan
02
Long horizons, with recovery
A planner chains them into long tasks on a real robot, and plans again when a skill does not do what it should.
03
Any state representation
The same learning works from object states, point clouds and camera images.
Method
An agent in the loop
-
Propose
An LLM agent proposes the effects of a concept: after which actions it becomes true, and after which it stops being true.
-
Learn
A neural classifier learns the concept from the demonstrations' image features, supervised by those effects. An inconsistent proposal cannot be learned, and its loss stays high.
-
Revise
The loss and the checks go back to the agent, which revises the proposal until the concept separates cleanly.
Derived concepts
Some concepts are built from others. No toy is left on the table when every toy is off it; a snack is clear when no bottle blocks it. UniPred learns these derived concepts too, and the planner relies on them: wiping needs the table free of toys.
NoToyOnTable(table) = ∀ toy. ¬OnTable(toy, table)
Results
Closed loop, in simulation
Two simulated versions of our real-robot tasks, on a Franka in MuJoCo. The robot perceives the scene from its camera, plans with the learned concepts, acts, and plans again when an effect did not happen.
Table cleaning
Put the toys in the box, push the box away, wipe the table, and bring the box and the sponge back.
Cluttered retrieval
Put every target in the tray, moving what blocks it first. When a target is still blocked, the robot replans.
In numbers
Closed-loop execution on the real robot
Fractional task success; ten test tasks, three seeds.
* Given the ground-truth predicates and partial ground-truth labels. The same perception and skills for every planner; the human oracle picks each next skill by hand.
Learning time against IVNTR
Total learning time, mean over five seeds.
| Domain | State | UniPred | IVNTR |
|---|---|---|---|
| Satellites | objects | 95.2 | 94.0 |
| Blocks | objects | 81.6 | 73.2 |
| Tools | point clouds | 100.0 | 100.0 |
| Packing | images | 100.0 | 0.0 |
| Table cleaning | objects | 93.4 | 0.0 |
Real robot
Long-horizon tasks on a real robot
Two real-robot domains, 20 evaluation tasks and 20 teleoperated demonstrations per domain. Every run is in a closed loop: after each skill the robot looks again and replans if needed.
Table cleaning
Put every toy in the box, then wipe the debris off the table with a towel. Wiping needs the table free of toys.
Three toys 3×
Three toy vehicles into the box, then the towel, then wiping. The complete run.
Two toys 3×
Another arrangement: two toys to put away before the towel can finish the job.
Cluttered retrieval
Put the target snacks in the basket. Bottles in the way have to be moved first.
Light clutter 4×
Few obstructions around the targets.
Moderate clutter 4×
A bottle moved out of the way before the target.
Heavy clutter 4×
Several obstructions cleared before the targets.
Failure recovery
When a skill does not have its expected effect, the robot sees it and replans from what it observes.
A complete run with recovery 3×
Table cleaning, through failed attempts, to the end.
Retrying a toy grasp 3×
The toy stays on the table after a missed pick; the robot picks again.
Retrying a towel grasp 3×
The towel grasp is retried before wiping.
Failure cases
Replanning needs correct concepts and reliable skills; it cannot recover from every perception or execution error.
Unsuccessful pick and place
If a concept misreads the resulting scene, the next skill may be the wrong one.
Failed object transfer
A valid plan cannot guarantee a grasp or a placement; errors may persist.
Incomplete wiping
The towel may not clean the table, even when the skill is retried.
Paper
Unifying Deep Predicate Invention with Pre-trained Foundation Models
Abstract
Long-horizon robotic tasks are hard due to continuous state-action spaces and sparse feedback. Symbolic world models help by decomposing tasks into discrete predicates that capture object properties and relations. Existing methods learn predicates either top-down, by prompting foundation models without data grounding, or bottom-up, from demonstrations without high-level priors. We introduce UniPred, a bilevel learning framework that unifies both. UniPred uses large language models (LLMs) to propose predicate effect distributions that supervise neural predicate learning from low-level data, while learned feedback iteratively refines the LLM hypotheses. Leveraging strong visual foundation model features, UniPred learns robust predicate classifiers in cluttered scenes. We further propose a predicate evaluation method that supports symbolic models beyond STRIPS assumptions. Across five simulated and two real-robot domains, UniPred achieves 2–4× higher success rates than top-down methods and 3–4× faster learning than bottom-up approaches, advancing scalable and flexible symbolic world modeling for robotics.
BibTeX
@article{wang2026unipred,
title={Unifying Deep Predicate Invention With Pre-Trained Foundation Models},
author={Wang, Qianwei and Li, Bowen and Luo, Zhanpeng and Xu, Yifan and Kamat, Vineet and Menassa, Carol and Gray, Alexander and Silver, Tom and Scherer, Sebastian and Sycara, Katia and Xie, Yaqi},
journal={IEEE Transactions on Robotics},
year={2026},
pages={1--20},
doi={10.1109/TRO.2026.3742972}
}