Explainable AI, or better said, interpretable Machine Learning supports us in understanding the decisions made by our complex models, when and why the model might fail or succeed, and how we can improve our model’s performance by having gained more insight into its learning structure and the input data. Here at AIT, we incorporate explainability in our AI systems to improve the performance and detect biases in the data (e.g., XAI helps to detect gender biases in NLP models). This information develops and advances trust and fairness in deploying ML-models in the industry and provides ML-practitioners a guideline on the need for further research on transparent algorithms.