Most banks adopting explainable ML expect a clean narrative: model makes decision, system explains it, regulator approves. The reality is messier.
Where the friction shows up
The first issue most risk teams hit is that SHAP values and LIME outputs are technically accurate but operationally confusing. A credit analyst reviewing a declined mortgage application does not need a bar chart of feature contributions — they need a sentence they can defend in writing to a customer. Translating model output into plain language requires a separate layer of work that vendors rarely scope into implementation timelines.
Regulatory pressure versus model performance
Canadian regulators under OSFI guidelines expect institutions to demonstrate model explainability at the individual decision level. This pushes many banks toward inherently interpretable models like logistic regression or gradient-boosted trees with constrained depth — not because those models perform best, but because explaining a deep neural network to an auditor remains genuinely difficult. Several mid-sized institutions have quietly rolled back neural architectures for this exact reason.
What actually works in practice
The implementations that hold up tend to share one characteristic: explainability was treated as a product feature from day one, not retrofitted after model training. Teams that built explanation templates alongside model development — defining which features needed natural-language descriptions before training started — shipped usable systems faster and with fewer compliance revisions. The technical tooling matters less than the workflow design around it.
Explainability in banking is a genuine operational requirement, not a checkbox. Getting there takes more process discipline than most technology teams anticipate going in.