Same Financials, Different Identity: A Counterfactual Audit of Demographic Sensitivity in LLM-Based Credit-Risk Scoring

Large language models (LLMs) are being explored as decision-support tools for financial risk assessment. Their outputs may, however, respond to demographic descriptions that are unrelated to the underlying financial record. We study this behavior through a paired counterfactual audit. The same 100 commercial credit and cheque-payment profiles are evaluated repeatedly, with only one demographic descriptor changed at a time. The experiments cover 7 model variants, three prompting strategies, and descriptors related to gender, education, and citizenship. While paired gender score differences remain within a ±1.5 point equivalence margin in 12 of 21 model–strategy cells, education shows a strong, unambiguous effect. Risk decreases monotonically with education level (103 of 126 pairwise contrasts remain significant after Holm correction, score differences reaching 26.0 points between primary-school and doctorate descriptors), it flips up to 42% of approve/reject decisions, and elicited rationales name education as a risk factor even though education is not a financially legitimate predictor. Citizenship produces larger shifts still, flipping up to 51% of decisions while leaving AUC almost unchanged in most cells, and in one model weakening ranking (0.794 to 0.438). Similar to education, rationales name the descriptor as the reason. Unlike citizenship, whose jurisdictional cues can legitimately enter sanctions or AML/CFT screening, the education effect has no such justification, making it the clearest evidence of demographic bias. A follow-up experiment in English, Chinese, and Arabic shows that citizenship sensitivity depends on both prompt language and prompting strategy, affecting the magnitude and direction of score shifts. The study provides a standardized counterfactual framework for auditing LLM-generated scores before they are considered for high-stakes financial workflows.