News

New Transformer-Based Model "NSODE" for Learning Dynamical Laws from Data

Researchers have made progress in machine learning (ML) applications to discover natural laws solely from observational data. To further this endeavor and infer dynamical laws efficiently, a transformer-based sequence-to-sequence model has been developed - the Neural Symbolic Ordinary Differential Equation (NSODE).

Sören Becker, a PhD student, together with Michal Klein, a former MSc student, both in the group of Helmholtz AI YIG leader Niki Kilbertus, and team have made significant progress in machine learning (ML) applications to discover natural laws solely from observational data. To further this endeavor and infer dynamical laws efficiently, the researchers have developed a groundbreaking transformer-based sequence-to-sequence model called Neural Symbolic Ordinary Differential Equation (NSODE).

The NSODE model is specifically designed to recover scalar ordinary differential equations (ODEs) in symbolic form from irregularly sampled and noisy observations of a single solution trajectory. Unlike many existing methods, NSODE excels at directly inferring the underlying dynamics from the observed data, making it more adept at accurately recovering ODEs. The symbolic representation of the inferred law offers notable advantages, as it is both parsimonious and interpretable, facilitating analytical analysis. The process involves training NSODE in a supervised manner using encoder-decoder transformers to map observed trajectories, directly to symbolic equations as strings.

The team conducted extensive empirical evaluations, demonstrating that NSODE performs better or on par with existing methods in terms of accurate recovery across various settings. Moreover, the model's efficiency is remarkable; after pretraining on a large set of ODEs, it can efficiently infer the governing law of a new observed solution in just a few forward passes.

To ensure robustness and comprehensiveness, the researchers generated a substantial dataset of scalar, autonomous, non-linear, first-order ODEs, along with numerous numerical solutions derived from various random initial conditions. All solutions underwent careful checks for the convergence of numerical integration to ensure high-quality data.

The release of NSODE advances the field of ML-driven scientific discovery, providing scientists in the natural sciences with a powerful tool to extract dynamical laws from observational data efficiently. With NSODE's outstanding performance and interpretability, it holds the potential to accelerate research and understanding in diverse scientific domains.

Learn more about NSODE and the paper here: https://proceedings.mlr.press/v202/becker23a/becker23a.pdf

This research opens up exciting possibilities for further advancements in symbolic regression and the direct inference of dynamical laws, inspiring hope for new breakthroughs in machine learning applications across the sciences.

The work has been published in Proceedings of the 40 the International Conference on Machine Learning in Honolulu, Hawaii, USA.