Robot Programming by Demonstration with Situated Spatial Language Understanding

Abstract

Robot Programming by Demonstration (PbD) allows users to program a robot by demonstrating the desired behavior. Providing these demonstrations typically involves moving the robot through a sequence of states, often by physically manipulating it. This requires users to be co-located with the robot and have the physical ability to manipulate it. In this paper, we present a natural language based interface for PbD that removes these requirements and enables hands-free programming. We focus on programming object manipulation actions-our key insight is that such actions can be decomposed into known types of manipulator movements that are naturally described using spatial language; e.g., object reference expressions and prepositions. Our method takes a natural language command and the current world state to infer the intended movement command and its parametrization. We implement this method on a two-armed mobile manipulator and demonstrate the different types of manipulation actions that can be programmed with it. We compare it to a kinesthetic PbD interface and we demonstrate our method’s ability to deal with incomplete language.

BibTeX Entry

@inproceedings{forbes2015icra,
  title = {Robot Programming by Demonstration with Situated Spatial Language Understanding},
  author = {Forbes, Maxwell and Rao, Rajesh P.N. and Zettlemoyer, Luke and Cakmak, Maya},
  year = {2015},
  booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
  pages = {2014--2020},
  doi = {10.1109/ICRA.2015.7139462},
  isbn = {978-1-4799-6923-4},
  type = {conference},
  video = {https://www.youtube.com/watch?v=_CUSX1GhjuA}
}