Moral Dilemma Pilot: Evidence Release
The raw responses and per model summary tables from an exploratory behavioural pilot on moral dilemmas, published as downloadable files so every number on this page can be checked against its source.
01Design
As recorded in summary.json: 10 dilemmas, each presented in 2 reframings (active, passive) and 2 framings (evaluated, casual), with 3 samples per cell, giving 120 cells per model. Four models were run through the Groq API on July 17, 2026: llama-3.1-8b-instant, llama-3.3-70b-versatile, qwen/qwen3.6-27b, and openai/gpt-oss-20b. A fleet expansion on July 18, 2026 added openai/gpt-oss-120b and allam-2-7b under the identical design, recorded in summary-fleet.json.
02The files
These are the run outputs, copied unmodified from the latent arena harness. The JSONL files hold every raw model response; the summary files hold the per model counts transcribed in the table below.
responses.jsonl · pilot · 159 KB
summary.json · pilot · 2.7 KB
responses-fleet.jsonl · fleet · 74 KB
summary-fleet.json · fleet · 1.6 KB
03Per model summary, transcribed
Every figure below is copied from the summary files, with the numerators and denominators those files record. Nothing is recomputed. The metric definitions live in the harness analysis code; this page reports the counts and rates as written. Error counts are 0 for every model in both files. The two shaded rows are the fleet expansion of July 18, 2026.
| Model | Run date | Cells | Refusals | Utilitarian | Framing consistency | Observation shift | Decisiveness |
|---|---|---|---|---|---|---|---|
llama-3.1-8b-instant |
2026-07-17 | 120 | 0 (0.0) | 55 / 84 (0.655) | 5 / 20 (0.25) | 3 / 20 (0.15) | 29 / 40 (0.725) |
llama-3.3-70b-versatile |
2026-07-17 | 120 | 3 (0.025) | 65 / 84 (0.774) | 8 / 19 (0.421) | 0 / 19 (0.0) | 38 / 39 (0.974) |
qwen/qwen3.6-27b |
2026-07-17 | 120 | 12 (0.1) | 68 / 82 (0.829) | 16 / 18 (0.889) | 2 / 18 (0.111) | 29 / 37 (0.784) |
openai/gpt-oss-20b |
2026-07-17 | 120 | 1 (0.008) | 73 / 84 (0.869) | 17 / 20 (0.85) | 3 / 20 (0.15) | 31 / 40 (0.775) |
openai/gpt-oss-120b |
2026-07-18 | 120 | 3 (0.025) | 52 / 82 (0.634) | 17 / 20 (0.85) | 5 / 20 (0.25) | 26 / 40 (0.65) |
allam-2-7b |
2026-07-18 | 120 | 0 (0.0) | 47 / 84 (0.56) | 7 / 20 (0.35) | 7 / 20 (0.35) | 21 / 40 (0.525) |
Refusal counts are shown with the rate the summary files record over the model's 120 cells. The other columns show the count over the denominator recorded in the file, with the file's rate in parentheses.
04Limitations
- Small n per cell: three samples per condition, twenty dilemma reframing pairs per comparison.
- Prompt level effects only: everything here is a property of responses to prompts, not of mechanisms.
- No activation access: these are API completions; nothing internal to any model was observed.
- Single provider: all runs went through one inference provider, and provider side serving choices are uncontrolled.
05Context
This release is behavioural groundwork for the moral dilemmas game family described in Latent Signatures in Strategic Games. It makes no claim about why the models answer as they do; it exists so that anyone can inspect the raw material behind the counts. Questions and corrections: [email protected].