A more efficient training workflow for hybrid modeling

A new generation of machine learning (ML) models brings together the established laws of physics with algorithms, optimization, and data-driven inferences. Defined as hybrid models, they have the potential to take the best features from physics-based models and ML models while addressing the shortcomings in each approach.

Researchers at Carnegie Mellon University are developing more robust and efficient methods for training, simulating, and optimizing hybrid models to advance the emerging field of scientific machine learning.

Engineers using first principles to model systems in energy, healthcare, and other fields must make simplifying assumptions. "Our first-principles models are indispensable, but in practice they can be incomplete," says Victor Alves. "The data aren't always fully explained with existing models and assumptions."

Machine learning can bridge this gap with its ability to capture unknown behavior using data alone. Although it excels at identifying complex patterns in data, machine learning has some clear limitations. It may require a lot of data that we do not have, and it does not extrapolate well beyond the training domain. In addition, machine learning models are opaque and may lack interpretability and insight.

Hybrid models position first principles and machine learning as complementary, not competing, approaches. "We shouldn't throw away all of the knowledge that scientists and engineers have developed over centuries, just to replace it with machine learning," says Alves, a postdoctoral fellow in the Department of Chemical Engineering. "We should use machine learning to augment that knowledge and fill in the gaps." Hybrid models combine the best of both worlds. They often require less data than standard machine learning models and can better extrapolate beyond the data used for training.

Applying the simultaneous approach

To create a training workflow for hybrid dynamic models that integrates machine learning, first principles, and physical constraints, chemical engineering Ph.D. student Laurens Lueg collaborated with Alves, Carl Laird, Daniel Schicksnus, John Kitchin, and Lorenz Biegler. "Constraints are important in engineering because they are a mathematical way to translate safety limits, performance requirements, and environmental regulations," says Alves.

Their training workflow uses the so-called simultaneous approach for dynamic optimization, developed years ago by Biegler, who is the Covestro University Professor in the Department of Chemical Engineering. This approach can be more efficient since it converges the model equations simultaneously with the training problem, instead of repeating model simulations over and over. By applying long-standing methods to new technology, the researchers were able to fine-tune the training algorithms so that they can be solved more efficiently for key engineering applications.

One of the case studies used to demonstrate proof of concept is a bioreactor. The biopharmaceutical industry uses bioreactors to produce drugs from living cells. Some of the chemical reactions that happen inside these bioreactors are not well understood or are too complicated to model efficiently. With a hybrid model, however, engineers can bring together known physics, like principles of mass and energy conservation, with machine learning to generate a mathematical model that can represent the reaction kinetics.

When Lueg, Biegler, and their collaborators tested their methods on case studies from different applications, they took a systematic approach, formulating the problem statement the same way each time. Their training approach is general, and it can be used to build more reliable models in fields beyond classical chemical engineering applications, such as energy, sustainability, and biopharmaceuticals.

For media inquiries, please contact Lauren Smith at lsmith2@andrew.cmu.edu.