The One-Word Census: Answer-Choice Conformity Across 44 Language Models

Methodology: The One-Word Census Protocol

• Uses 31 deliberately minimal single-turn prompts, each naming a category with many valid one-word answers (e.g., "Name a tree.", "Name a color.");
• Each model answers each prompt four times, with no system prompt;
• Evaluation uses exact-match on normalized tokens—no embeddings, no human judge;
• Cost ≈ $1 per model; full implementation reported in arXiv:2607.12796v1.

Key Findings

• Convergence is pervasive and extreme: in 7 of 31 categories, a single answer accounts for >80% of all model outputs (e.g., "Pick a word — any word." → "serendipity" at 41%);
• Model-level conformity varies substantially: the answer-choice surprisal (leave-one-out average −log₂ probability of a model’s answers under the pooled distribution of all other models) differs by >4× between highest- and lowest-conformity models;
• Variation is structured: persona-tuned (e.g., Claude-instant-1.2) and community-tuned models (e.g., OpenAssistant, zephyr-7b-beta) deviate most strongly, while base models (e.g., Llama-2-7b-chat, Gemma-2b) and instruction-tuned models (e.g., Mistral-7b-instruct) align more closely with the population.

Methodological Contribution

• Introduces the One-Word Census—a low-cost, highly reproducible, annotation-free behavioral probe for language models;
• Reveals that model output distributions are not uniform samples but reflect structured biases shaped by training data priors and alignment strategies;
• Provides a new axis for interpretability, safety alignment assessment, and community preference modeling: answer-choice surprisal serves as a quantifiable metric of a model’s ‘consensus deviation’.