Guest Contributor
As machine learning enters the mainstream, consumers may assume that it can solve almost any problem. This is not true, says Bruce Lee, a doctoral student in Penn Engineering’s Department of Electrical and Systems Engineering. Lee’s research works to identify how robotic systems learn to perform different tasks, focusing on how to tell when a problem may be too complex — and what to do about it.
Lee, who is advised by Nikolai Matni, Assistant Professor in Electrical and Systems Engineering and member of the Penn Research in Embedded Computing and Integrated Systems Engineering (PRECISE) Center, studies how robotic systems learn from data, with the goal of understanding when robots struggle to learn a dynamic system, and what approaches might be effective at combating those challenges.
His work offers insights into the fundamental limits of machine learning, guiding the development of new algorithms and systems that are both data-efficient and robust.
“When I try to apply a reinforcement learning or imitation learning algorithm to a problem, I often reach a point where it does not work, and I have no idea why,” says Lee. “Is it a bug in my code? Should I just collect more data or run more iterations? Do I need to change the hyperparameters? Sometimes, the answer is none of the above. Rather, the problem is impossible to learn effectively, no matter what learning algorithm I use. My work can help researchers understand when this is the case.”
Improving the way robotic systems learn from data enhances the safety and efficiency of self-driving cars, enabling them to make more reliable decisions in complex, dynamic environments. Similarly, robots operating in human environments, such as in health care or manufacturing, can become more adaptable and capable of performing a wider range of tasks with minimal human intervention. Ultimately, the goal is to create robotic systems that can better serve humanity, contributing to advancements in various fields including transportation, health care and beyond.
“I was drawn to this research area out of a deep fascination with the potential and limitations of machine learning in solving complex problems, particularly in robotics,” says Lee. “The dynamic nature of real-world environments presents a unique challenge for robotic systems, and I saw an opportunity to leverage my background in control theory and statistics to make a significant impact. My motivation stems from a desire to bridge the gap between theoretical machine learning models and their practical applications, especially in situations where safety and reliability is paramount.
One case study that Lee is currently considering is a project by Google that aims to help robots learn general control policies from data. The generalist policies are intended to help robots perform new tasks with a limited amount of training data by leveraging similarities to tasks that have been conducted during the training phase.
“While these methods have been shown to be effective through a collection of experiments, they clearly have severe limitations,” says Lee. “In particular, if you try to deploy the controller on a new task that is not ‘sufficiently close’ to what was seen during training, the controller will behave unpredictably. The challenge is that ‘sufficiently close’ is a very loosely defined notion. Our goal is to apply tools from statistical learning theory and control theory to characterize exactly how similar the new task needs to be to training tasks in some simplified, analytically tractable settings.”
“New results in machine learning, such as ChatGPT, Midjourney, diffusion models or deep learning in general, are very exciting and are enabling new capabilities we haven’t seen before,” says Matni. “However, despite this exciting progress, they are still unreliable and data-hungry. While this is not a problem when applied to chatbots or image generation, it can be catastrophic when applied to safety-critical systems that interact with the physical world, such as self-driving cars.”
By understanding the statistical properties of learning models and controllers from data, Matni adds, “We can understand how to build systems for which learning is provably efficient, robust and safe, and importantly avoid systems and scenarios where it is provably expensive and unreliable.”
When the Problem Is Too Hard
One key takeaway from the research, Lee says, is that sometimes the problem is just too difficult. Control system engineers and researchers often think their job is to design an effective control system for a specific system facing a specific challenge, but this isn’t always the right approach.
“Sometimes no matter what they do, they are doomed to failure,” says Lee. “Instead of changing the parameters of the learning algorithm or collecting more data, it might be better to think about how to make the problem easier by adding an actuator or improving sensor placement.”
One of Lee’s recent studies focused on a common control system design called a linear quadratic regulator. He found that the amount of data required to learn the linear quadratic regulator from offline data sets can grow significantly as the problem becomes more complex. “This is a surprising result, because it illustrates that even for one of the most commonly used control approaches, learning can be prohibitively data-hungry,” says Lee.
In addition to helping researchers understand when their problem is too hard, Lee’s results can help to guide the design of systems that are easier to control.
Lee, who is expected to graduate in 2025, is also studying how researchers and practitioners can work around the fundamental limits of what robotic systems can do. One approach to doing so is strategically designing systems to make them as easy to learn as possible. Another is to supplement the data collected from any system of interest with data from related systems, leveraging the similarity between the two to continue to learn while using less data from the system of interest.
“Many outside the field think machine learning can solve almost anything. My work helps to show that it cannot,” says Lee. “In instances where we require robots to learn from interacting with the environment, such as in autonomous driving or robotic manipulation, collecting interactions can be extraordinarily resource intensive. Our results show that if we have complicated systems with a high number of states, then learning an adequate control system from scratch will require an exorbitant amount of data to be collected from the world, which may be impossible for physical robotic systems.”
This story was written by Liz Wai-Ping Ng, from the PRECISE Center.