Mis-Alignment in AI4Science

Mon, 14 Sep 2026 00:00:00 +0000

Numerical solutions to partial differential equations (PDEs) are the bedrock of scientific computing. They model a wide range of physical phenomena, from fluid dynamics to electromagnetism, from quantum mechanics to biology, from climate modeling to astrophysics. However, these methods tend to be computationally expensive when we solve such equations at scale, and still being only an approximation determined by the nature of chosen discretization, specific solver, and convergence rates dictaed by the stiffness of such equations.

Neural emulators, trained on rich and quality simulation data have emerged as a faster way to approximate the solutions, although they often lack guarantees to be accurate or reliability beyond what the training data provides- specific regime, parameter space, and initial conditions. The imperative question is to ask if the neural emulators ensure that it will generalize well to unseen scenarios beyond training when deployed. The question is how do we know if the neural emulator is aligned with the true solution of the PDE, and how do we ensure that it will generalize well to unseen scenarios?

This is taken further by a scientific foundation model, which is a large neural network trained on a wide range of scientific data and tasks. The goal of a scientific foundation model is to learn a general representation of the underlying physics that generalizes to a wide range of scientific problems. This is similar to how large language models (LLMs) are trained on a wide range of text data and can be used for a wide range of natural language processing tasks. A model that can be used for a large class of problems that share similar mathematics, conservation laws, are bounded and time evolving.

Fun What makes a Scientific foundation model different from the LLM foundation models is that they are not be studied for emotions and consciousness :). So such discussions are not relevant to the scientific foundation model.

Chain-of-thought monitorability in AI models

COT became a popular technique to improve the performance of large language models (LLMs) on complex reasoning tasks. The model is asked to generate intermediate reasoning steps before producing a final answer which allows the model to break down complex problems into smaller, more manageable parts, and can lead to more accurate and reliable answers. AI developers and researchers have used of chain-of-thought for monitorability to understand the decision-making process of AI models. (rRecent research has shown that some models, such as Astra, have a significant decrease in chain-of-thought monitorability compared to previous models. This means that the model is better at performing tasks without verbalizing its thought process, which can make it more difficult to monitor and control its behavior.)

Mis-Alignment in AI4Science

The field of AI4Science has seen significant advancements in recent years, with the development of powerful machine learning models and techniques that have been applied to a wide range of scientific problems. However, there is a growing concern about the misalignment between the goals of AI researchers and the needs of the scientific community. This misalignment can lead to the development of models that are not well-suited to the specific challenges of scientific research, and can result in wasted resources and missed opportunities for scientific discovery.

Interpretability

Frontier labs and researcher have been exploring the use of interpretability methods to understand the decision-making process of neural networks in scientific applications. These methods can help us visualize and interpret the learned representations in neural networks, and investigate the role of feature interactions in scientific datasets and how they affect model predictions.

Interpretability x AI4Science

Possible directions to explore:

Understanding the decision-making process of neural networks in scientific applications.
Developing methods to visualize and interpret the learned representations in neural networks.
Can we interpretabiity be used to identify biases or errors in scientific datasets, such as those found in reanalysis products like ERA5?

https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2026JH001387 https://arxiv.org/pdf/2603.29658

Physical interpretability

https://arxiv.org/pdf/2604.22580v1 https://arxiv.org/pdf/2512.24440

Going Beyong Training and History

AI models, largely trained on many physics based simulated data and observations ( or some combination of both) seem more capable and accurate, efficient than the physics based model. But the real question is how are we going to predict if they are aligned to our long term objectives of the true phenomenon rather than just the training distribution.

Bad predictions from AI based models in future are of catastrophic consequences. Although they may not be a wedspread situation and assessed during training and testing, real world is more complex, doesn’t guarentee that the future has signals which are statistically similar to the past, the distributions represented by the historical data will also hold in the future. Although there are several methods to address such challenges, they are yet far to establish guarentess and understanding that physics based models produce.

Using Model’s activation space to estimate the probability of particular output of behaviour is crucial.

A lot of interesting research areas are laid down here for reference: https://coefficientgiving.org/tais-rfp-research-areas/#17-white-box-estimation-of-rare-misbehavior

They ask for a proposal with certain criteria to evaluate the work and benchmark.