AI concepts
Explainable AI / XAI (the explainability of AI model decisions)
A set of techniques that make it possible to understand why an AI model made a specific decision. Required by the AI Act for high-risk systems, by the financial sector through the KNF Recommendation, and by the medical sector through the MDR. Without XAI, a model is a black box that is not acceptable in regulated contexts.
Primary source: DARPA XAI Program 2017-2021, NIST Four Principles of Explainable AI 2021, IBM Research AI Explainability 360 Library 2023
DARPA launched the XAI programme in 2017 in response to growing reliance on ML models in critical applications. Four years later the programme produced libraries, metrics, and a taxonomy that became an industry standard. In 2021 NIST published Four Principles of Explainable AI, which became a reference point in regulations.
The four NIST principles
Explanation. The system provides an explanation together with the decision. Not just "credit denied", but "credit denied due to factors X, Y, Z, in that order of weight".
Meaningful. The explanation is understandable to its target audience. An explanation for a client differs from an explanation for a regulator.
Explanation Accuracy. The explanation genuinely reflects what the model did. Post-hoc rationalisation is not XAI.
Knowledge Limits. The system indicates when it does not know or is not certain. No feigning of confidence.
Three classes of technique
IBM Research AI Explainability 360 Library from 2023 standardises implementations.
Inherently interpretable models. Decision trees, linear regression, generalized additive models. The model's structure is itself the explanation. The safest for critical applications, but lower prediction quality than neural networks.
Post-hoc model-agnostic methods. LIME (Local Interpretable Model-agnostic Explanations), SHAP (SHapley Additive exPlanations). They work on any model and explain a decision for a specific case through local approximation. The most popular in enterprise in 2025.
Attention visualisation for LLMs. A visualisation of which fragments of the input the LLM "looks at" while generating a specific answer. Useful for debugging, limited as a genuine explanation (Stanford research from 2023 questions the reliability of attention as a proxy for a model's decision).
Polish context
Recommendation S of the Polish Financial Supervision Authority from 2023 requires banks using credit scoring models to provide explainability documentation at the level of an individual client decision. So a client who is declined must receive the factors behind the decision, not just the fact of the refusal.
The AI Act from February 2026 extends this requirement to all high-risk systems. Penalties for the absence of XAI in a regulated area: up to 35 million euro.
The cost of implementing XAI
Adding SHAP to an existing model: 50 to 150 thousand zloty one-off (engineering, testing). Inherently interpretable models from the start: no additional cost, but typically 5 to 15 percent lower prediction quality. The choice is a trade-off between accuracy and explainability, dependent on context.
We make the XAI strategy decision for planned deployments within the AI Readiness Audit.