What is the purpose of the diagonally-masked self-attention (DMSA) mechanism in SAITS, and how is its effectiveness validated in the ablation study?

The diagonally-masked self-attention (DMSA) mechanism prevents each time step from attending to itself during attention computation by setting the diagonal entries of the attention matrix to negative infinity before softmax, making those self-attention weights effectively zero. This is intended to stop a missing position from using its own partially estimated value when imputing missing values. Its effectiveness was validated in an ablation study that compared SAITS-base (with diagonal masks) against SAITS-base-w/o (a variant with the masks removed), and SAITS-base consistently performed better across all datasets, confirming that the diagonal masking improves imputation capability.

In SAITS, DMSA is a core building block. Formally, the query and key product QK^T is passed through a diagonal mask before scaling and softmax, where DiagMask(A)_i,j = -infinity if i = j and A_i,j otherwise. After softmax, diagonal weights become zero, so a time step cannot incorporate information from itself when estimating missing values. This self-reference avoidance is important because the model is trained to reconstruct missing entries from observed context rather than copy its own input values. DMSA is implemented in multi-head attention, and SAITS stacks two DMSA blocks followed by a weighted combination block. The ablation validation is described for SAITS-base, which is stripped of some full SAITS components to isolate DMSA's effect. The study compares SAITS-base with SAITS-base-w/o, where the latter removes the diagonal masks from the self-attention layers. Because SAITS-base consistently outperforms SAITS-base-w/o across all tested datasets, the authors conclude that adding diagonal masks to self-attention enhances the model's imputation performance.