Philosophical Commitments to LLM Evaluations: The Problem of Moving Goalposts and Observational Relativity

While most technical and philosophical research on LLMs tries to find a translation from human functions to computational principles, understanding the flip side — how computational principles translate into natural ones — is often overlooked. This can be partly attributed to the lack of a general awareness of the capabilities and limitations of LLMs.  We can leverage these systems for mastering language in its many dimensions but remain fundamentally unable to describe the current and predict the future functionalities of this technology. I argue that this uncertainty stems from our inability to evaluate these systems holistically, which, in turn, is rooted in two interrelated challenges: the fluid philosophical definitions of intelligence and our own observational bias. Seemingly rational and grounded benchmarks for LLM evaluations are provided and broken iteratively as definitions of intelligence are not fixed but reflect the evolving state of societal and technological development. Observational bias further blinds us to the capacities of non-human systems. Together, these factors contribute to what I term the Evaluation Problem, a profound inability to map and measure the cognitive capacities of non-human systems. Through a diachronic analysis of intelligence definitions, I illustrate these issues and refute two prominent counterarguments: that externalist benchmarks suffice entirely and that mechanistic interpretability offers a complete solution. Finally, I explore the implications of these challenges for the future of intelligence evaluation.