Everything below is being written word-by-word, by a human( me) to make sure my understanding is not shortcircuited by AI.

Introduction: The game of ‘who is the best?’

In statistical learning theory and traditional machine learning, we learn that the empirical risk minimization is nothing but an expectation wrt the true data distribution.

$$E_{p(\hat y ,y)} [l(\hat y,y) ]= \sum_{k} (\hat y_k-y_k)^2$$

where, $p(x,y)$ is the probability distribution and the model output $ \hat y = f_{\theta}(x).$

The choice of loss function $l$ which measures the discrepancy of the model output from the true targets is equally important. Now consider a metric that is not able to distinguish between two probability distributions $p$ and $q$.

$$D(p, q)=0 \implies p=q$$

If maybe that sometimes for some p close enough to q. ELBO uses this idea for variational methods in deep learning by parameterising the distribution $p$ as $p_{\theta}$ and then minimizing bounds on KL.

Convergence and sequences:

Assume we have an iterative algorithm that furnish the parameters $\theta$ coming from a parameterized function or a neural network. As $n$ is a finite number of iteraions that bring us closer to the true distribution, it maybe that our metric is not able to read the discrepancy beyond a small value. This makes our metric not useful beyond that threshold and everything beyond it, it remains blind. Hence such an algorithm or procedure that relies on this metric will also behave in a similar fashion. Our choice of the metric is thus imporant not only at the end, but it also guides the iterative steps of our algorithm itself.

One such example is a metric called RMSE which becomes blind and causes smooth and blurry outputs( in models where we predict fields on a grid.) Another example is KL divergence.

A madeup exmaple is if one want to distinguish two distributions and the distance only depends on the mean. Then such a metric fails to distinguish between a Gaussian, a uniform distribution whose support is summetric about $0$, or anything that have the same mean. As you can see, this is why we will never make one.

$$ $$

Evals Saturation: When evaluation noise exceeds true peformance gap

Primary reason why benchmarks in LLMs are useful is that they help us decide the strength of one model over the other. But When AI Benchmarks Plateau, we are looking at a scenarios where a lot of them actually start to look similar in nature. Several benchmarking datasets exist that measure how good an LLM specific to different categories. When all models start to get very close to each other, they loose meaning and their distinguishing property.

The value of such benchmark is then lost and it stops measuring any decisive contribution to decisions down the line. If is all sounds complicated, it just means that our measuring scale has divisions larger than the object we are trying to measure.

If you want to know some examples of AI Benchmarks- Math-500, LiveBench, LiveCodeBench, TruthfulQA and Humanity’s Last Exam.

Error models

Measurement resolution versus capability resolution. Since these models are stochastic in nature, every independent test will yield slighly different score and is caused by the evaluation noise. So seperating this noise from the true signal which the capability of the model needs to be filtered. I now come to a rough sketch of what a metric is before I go back to LLM world in discussions. These are some fundamental ideas in estimation theory. When we do linear regression, we are actually assuming a model for the error -

When we peform Ridge regression, our error model is. All models try to fit to the data by understanding what is the structure of the error and then accordingly distinguish noise from signal.

Joint Analysis of error models:

beyond the current paradigm in AI Benchmarks, focusing on differet granularity, multidimensional nature of tasks, increasing evaluation resolution etc could help us.

Resources : interviews

https://www.youtube.com/watch?v=XoGvCBRnwLs Umar Jamil has a lot of great content on his channel.

Resources : papers

Topic Wise

On Tokenization: A short video on why we use subwords, here.

Jobs

ARC: Alignment Research Center is hiring for a few positions. I am not affiliated with them, but I think they are doing some interesting work in the field of AI Alignment. If you are interested, check out their job postings. https://jobs.lever.co/alignment.org/617488b1-d742-4990-a037-b7f0e2ba68c9