Is LLM's Moral Decision Compositional?

arXiv:2606.11232 · cs.CL, cs.AI · Submitted 2026-05-29 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Is LLM's Moral Decision Compositional?".

Jane: Here's the gist of the paper. The gist The MORAL TROLLEY ARENA introduces a two-stage blind ELO benchmark for measuring how LLMs compose moral evidence, shifting moral auditing from isolated act rankings to judgments over composed moral evidence.

Jane: The single-scene arena first calibrates individual moral acts from a 229-scenario corpus across five Moral Foundations Theory foundations and assigns act-level ELO scores Each atomic moral act is evaluated alone before composition >.

Paper summary: Tom: So looking at the authors Zhang and Chen and Xiao et al., the title "Is LLM's Moral Decision Compositional?" points directly at this idea that their two-stage benchmark tests composition rather than just isolated preferences >

Jane: It’s about moving beyond simply ranking which single moral act a model prefers, to understanding how it combines those signals when multiple acts are in play >

Lu: The implication is that we need new benchmarks that probe these composition rules, not just measures of individual act rankings >

Meng: From a production standpoint, if we understand this compression and anchoring behavior they found with the intensity grid, we might be able to build better safety guardrails for how AI agents make complex decisions >

Lalam: I'm pretty sure the paper’s main point is that isolated foundation ranking doesn't reveal how models trade off multiple moral signals once those acts are composed >

Tom: Exactly. The paper shows that single-scene foundation ranking measures only isolated-act preference and misses a systematic premium on extreme acts when they are composed >

Jane: So, the big picture here is that for AI auditing, we need to measure the composition rules for composed moral evidence rather than just looking at isolated act rankings >

Conclusion: Tom: So we’ve been looking at these experiments that test whether an AI’s moral judgment is just a bunch of isolated choices or if it actually understands how to put those choices together.

Jane: The paper, "Is LLM's Moral Decision Compositional?", it really asks that question directly about the models.

Lu: It sets up this whole two-stage benchmark. First, they calibrate individual moral acts across different foundational theories—like Loyalty or Care—and then they test how those calibrated acts combine into a composite judgment.

Meng: From what I’ve seen, this is interesting because it shifts the focus from just ranking one good thing versus one bad thing to looking at the whole package.

Lalam: The core finding is that while the overall judgment of a combined scenario often follows what you expect from adding up the parts, there's a specific way it handles intensity.

Tom: So what does that compression look like in practice? What’s the actual mechanic they found when those acts are put together?

Jane: They found that even when you have matched component strengths, there's an anchoring effect where one extreme virtue with a minor wrong beats two smaller virtues in every model they tested.

Lu: That suggests the composition rule isn't just simple addition; it’s nonlinear and has a specific bias toward extremity.

Meng: If that holds up when we look at things at scale, it changes how you think about safety guardrails for complex AI agents making high-stakes decisions.

Lalam: I’m pretty sure the real point is that single-act ranking just tells you what an AI likes about one thing in isolation; this shows how it actually trades off multiple moral signals.

Tom: So, we’ve seen the mechanics of the benchmark and where they land on these composition rules. Now we need to talk about what this means for how we actually audit these models in the real world.

Weijia Zhang, Ruiqi Chen, Yunze Xiao, Weihao Xuan

University of Illinois Urbana-Champaign · University of Michigan · Carnegie Mellon University · The University of Tokyo

cs.CL, cs.AI

Submitted: 2026-05-29

Updated: 2026-10-18

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 82/100

The gist: The gist The MORAL TROLLEY ARENA introduces a two-stage blind ELO benchmark for measuring how LLMs compose moral evidence, shifting moral auditing from isolated act rankings to judgments over

Key concepts

Single-act Evaluation: This stage scores individual moral acts separately based on five Moral Foundations Theory. It measures how a model ranks one specific action in isolation before considering any other actions or contexts.
Composite Judgment: This measures the model's preference when deciding between two people based on a combination of their acts. It tests whether the final moral judgment is derived logically from the individual scores of each act involved in the comparison.
Extremity Premium: The finding that one extreme virtue with a minor wrong beats two minor virtues indicates that models systematically prefer actions at the extremes rather than simple averages or additive combinations of moderate acts.

Terminology

Benchmark — 2× in this paper · explained in 2 episodes

Summary

The gist The MORAL TROLLEY ARENA introduces a two-stage blind ELO benchmark for measuring how LLMs compose moral evidence, shifting moral auditing from isolated act rankings to judgments over composed moral evidence.

How it works

The single-scene arena first calibrates individual moral acts from a 229-scenario corpus across five Moral Foundations Theory foundations and assigns act-level ELO scores Each atomic moral act is evaluated alone before composition >.
The composite arena then combines calibrated acts into two-act moral items over a controlled intensity grid and measures the resulting composite preferences.

Key Findings

** Across ten frontier models, composite judgments are largely predicted by component act strength, but the relation is consistently compressed rather than simply additive Models also show non-additive intensity anchoring, bounded foundation-specific residuals after component control, and highly convergent composite preference surfaces across providers. 1. We introduce MORAL TROLLEY ARENA, a two-arena blind ELO benchmark that shifts moral auditing from isolated act rankings to judgments over composed moral evidence by linking act-level calibration with composite moral judgment. 2. We characterize the composition rule itself >. Across ten frontier models, composite judgments follow a compressed linear function of component ELOs (β¯ = 0.862 < 1) and exhibit intensity anchoring, in which (+2, −1) profiles outperform (+1, +1) profiles despite matched component strength. 3. We isolate foundation-specific residuals after controlling for component strength, finding a bounded positive residual for Loyalty (+9.58) and a negative residual for Care (−9.22), while the remaining three foundations stay near the linear prediction. 4. We show that composite preference surfaces converge across providers (mean pairwise r = 0.939 over 160-dimensional compositeELO vectors), indicating that the elicited composition behavior is a shared property of current frontier models rather than a single-model artifact. **

Methodology Details

The benchmark involves two linked arenas:

Single-scene arena: This stage elicits an act-level foundation ranking through blind pairwise trolley battles, aggregating ∼1,713 battles per model to yield a per-scenario component ELO.
Composite arena: This stage combines calibrated acts into two-act moral items over a controlled 160-profile composite grid by crossing all 5 = 10 foundation pairs with the 16 non-neutral intensity configurations in the set of −2, −1, +1, +2.

Analytical Framework

The analysis extracts four signals relating the composite arena to the single-scene baseline:

(i) Compressed composition rule: Composite ELO is tested against a linear function of component ELOs Ec(A, B) = βm · Es(A) + Es(B) + αm + ε, where slope βm < 1 indicates compressed integration.

(ii) Matched foundation baseline and residuals: Residuals ε from the component-ELO model are aggregated by foundation across all composites containing that foundation to measure controlled foundation-specific trade-offs >.

(iii) Intensity anchoring: Departures from simple addition are tested by grouping composites by the sum s = kA + kB of their two component intensities, finding that s=+1 composite items outscore s=+2 items across all ten models.

(iv) Cross-model convergence: Pearson r between models’ 160-dimensional composite-ELO vectors is computed, showing a mean off-diagonal r of 0.939 across all 45 pairs, indicating frontier models converge on the full composite preference surface >.

Conclusion

Single-scene foundation ranking measures only isolated-act preference; it does not reveal how models trade off multiple moral signals once acts are composed. The MORAL TROLLEY ARENA suggests that moral audits should measure composition rules for composed moral evidence, not only isolated-act rankings >.

The paper is 2606.11232 and it matters because it provides a new benchmark to understand how LLMs combine multiple moral signals in complex choices rather than just ranking isolated acts. The gist The MORAL TROLLEY ARENA introduces a two-stage blind ELO benchmark for measuring how LLMs compose moral evidence, shifting moral auditing from isolated act rankings to judgments over composed moral evidence.

Improvements for AI systems

Improved moral auditing involves measuring composition rules for moral evidence, not only rankings over isolated acts. This allows for understanding how models combine several moral signals within the same option rather than just checking isolated acts.
AI systems can be evaluated on their ability to handle two-act moral items over a controlled intensity grid, testing whether non-additive intensity anchoring occurs, such as the finding that s=+1 composite items outscore s=+2 items by a cross-model mean of +35.0 ELO points.
Moral audits can be designed to isolate foundation-specific trade-offs by using the residual analysis, where Loyalty receives a positive residual (+9.58) and Care receives a negative residual (−9.22) after controlling for component strength.
Frontier models can be benchmarked for cross-model convergence by comparing their 160-dimensional composite ELO vectors, which shows that mean off-diagonal r across all 10 2=45 pairs is 0.939, indicating a shared property of current frontier models.
AI systems can be stress tested by identifying which moral signals substitute for or offset others by examining the residual premium matrix, such as finding that in the Loyalty/Care grid, MER(Loyalty → Care) = 1.57 means that one unit of Loyalty intensity has about the same marginal composite-ELO effect as 1.57 units of Care intensity.

Abstract

Moral evaluations of language models typically infer a model's values from its choices between single acts, yet in everyday life a person is judged on their history of good and bad acts. Such evaluations cannot capture how a model combines one person's acts. We test whether a model's moral judgment is compositional, that is, whether its judgment of a person follows from its judgments of each act. We introduce Moral Composition Arena, which asks which of two people is morally better to give Elo ratings to 229 human-validated acts across five Moral Foundations Theory foundations (act Elo) and to 160 controlled composite profiles, each a person described by two acts (composite Elo). Across ten frontier models and 29,134 choices, summed act Elo explains 73% of the variance in composite Elo, but the composition rule is not additive. With nearly equal mean summed act Elo, one extreme virtue with a minor wrong outranks two minor virtues in every model (mean +35.0 Elo), an extremity premium that survives a nonlinear baseline and human relabeling of act intensities. Single-act evaluations thus predict most of how these models judge people described by two acts, but miss a systematic premium on extreme acts.