Abstract:
Existing approaches to evaluating LLM ethical competence place too much emphasis on the verdicts—of permissibility and impermissibility—that they render. But ethical competence doesn’t consist in one’s judgments conforming to those of a cohort of crowdworkers. It consists in being able to identify morally relevant features, prioritise among them, associate them with reasons and weave them into a reasonable conclusion. We identify the limitations of existing evals for ethical competence, provide an account of moral reasoning that can ground better alternatives, and discuss the philosophical implications if LLMs ultimately prove to be adept moral reasoners.