WP5: Explainability

  1. 1

    Self-Attention for Model-Specific Explainability

    Leveraging self-attention mechanisms to inherently include explainability by relating elements along the input sequence. This addresses the opaqueness of deep learning models.

  2. 2

    Visualizing Attention Scores

    Producing visual representations by coloring specific signal intervals based on their attention scores, providing clinicians with intuitive feedback on model focus.

  3. 3

    Model-Agnostic Explainability

    Incorporating model-agnostic explainability techniques, independent of the ML model used, such as LIME (Local Interpretable Model-Agnostic Explanations) and SHAP (SHapley Additive exPlanations).

  4. 4

    Human-in-the-Loop (HITL) for Explanation Refinement

    Improving explainability with HITL techniques: human experts review and numerically score explanation quality, feeding into a Bayesian optimization process for hyperparameter adjustment.

Explainability is one of the most important parts of any machine learning design today. The problem with deep learning is that the developed models are mostly opaque. Using self-attention mechanisms allow us to include model-specific explainability capabilities, because self-attention learns a representation by relating elements at different positions along the input sequence. Attention scores can be used to expand interpretability of the model by mapping them back to the input signals.

In this regard, we aim to produce a visual representation by coloring the specific signal intervals with an intensity proportional to their associated attention scores, therefore providing the clinician with valuable and intuitive feedback about how the model is scoring the PSG.