Ai2 releases FairStress: 300 two-candidate decision scenarios across 29 domains, testing whether persuasion flips decisions differently by demographic group
Checking only whether the final choice matches hides bias; FairStress tests whether the same persuasive argument works differently depending on who it favors.
// Key points
- It contains 300 realistic two-candidate decision scenarios across 29 domains including hiring, healthcare, lending, education, housing and criminal justice.
- Identity is varied from none to a demographically coded name, implied identity and an explicit label; adversarial prefixes appeal to authority, consensus, historical disadvantage and more, each with a matched version for the other candidate.
- Across models including Olmo, Llama, Qwen, Gemma, Claude and the GPT family, and across training stages, the same persuasive arguments make models more or less likely to change decisions depending on which group they favor.
- The benchmark is available to download and is being built into Ai2's open evaluation workbench Olmo-eval.
Builder's takeThis bears directly on AI Interview: in resume screening, a single line like "leadership recommends this candidate" from either side can sway the model. I'll bring FairStress's hiring setup into my own eval set, swapping names and adding adversarial prefixes to the same resume to see whether the verdict holds, and have the model output only scores against the job requirements rather than free-form reversals.