CAREER: Enhancing Robot Physical Intelligence via Crowdsourced Surrogate Learning

This Faculty Early Career Development (CAREER) grant develops a novel way to enhance skilled manipulation of objects as performed by human and robotic hands. Challenges associated with dexterous object manipulation have been a bottleneck limiting the application of robots in many production and service applications. This project introduces a novel chopsticks-like robot and associated methods that will allow robots to gradually develop 'physical intelligence' (skills that can be transferred to novel situations) by learning from a large group of people, who are distributed across physical location and time. The project uses 'crowdsourcing' to obtain a large set of data from people using the chopsticks robot to guide a similar, remote robot through industrial and social activities. Sensorimotor control signals observed during teleoperation are parsed by a novel artificial intelligence controller to improve autonomous robot skills. An interesting feature of this controller is its ability to compose new object manipulation skills from parts of previously observed skills. The project develops a user training method that leverages the robot's acquired physical intelligence to actively guide and improve robot teleoperation skills of humans. The innovations promise to advance the national health, prosperity and welfare by improving dexterity of robotic and human manipulation skills in a variety of applications, including manufacturing, medical surgery, and home assistance. The project includes an educational component that engages pre-college and college-age students from diverse backgrounds.

This project will develop and test a novel focus on improving dexterous object manipulation skills performed by human and robotic hands. The project advances three novel ideas. First, the PI will use Amazon Mechanical Turk to implement a teleoperation system (user interface, user management system, and robot learning database) linking a group of human operators with remote robots. This approach, called 'Crowdsourced Surrogate Learning' (CSL), builds a database of sensor, motor, and machine vision data collected while a large number of people separately use a novel chopsticks-like robot to teleoperate a similar robot to conduct industrial and social tasks. These human subjects experiments will provide the data needed for subsequent robot skill learning and generalization using an approach called 'Robot Composite Learning' (RCL). RCL implements a discretized joint state space of the robot and the object being grasped/manipulated, along with a statistical inference technique, to compose new command signals - in real-time - enabling the two-fingered robot hand to learn never-demonstrated skills such as new, in-hand manipulations of small items. Finally, the PI will develop a 'Reciprocal Skill Induction' method that enables the CSL/RCL system to actively guide and improve the robot teleoperation skills of humans by providing haptic guidance using 'virtual fixtures' that fade as a novel skill is learned.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.