SAITS uses diagonally-masked self-attention (DMSA) blocks in a cascade arrangement: a first weighted combination fuses representations from the first two blocks into a learned representation, which feeds the third block, and a second weighted combination produces the final imputation. This architecture, together with a joint-optimization training strategy for imputation, yields large measured accuracy gains: 12% to 38% lower MAE than BRITS, and 2% to 19% lower MAE than a Transformer trained under the same joint optimization, using only 15% to 30% of the Transformer's parameters.

SAITS is a self-attention-based imputation network designed for multivariate time series. The source describes its architecture as a cascade-weighted arrangement of three diagonally-masked self-attention (DMSA) blocks linked by residual connections. The first weighted-combination layer fuses the representations from block 1 and block 2 into a learned representation that is passed to block 3, and then a second weighted combination of that learned representation with block 3's output forms the final imputation. The diagonal masks and the weighted combination strategy were validated in ablation studies; removing diagonal masks reduced performance, and the two-block imputation structure plus weighted combination were cited as key contributions. A joint-optimization training approach (referred to as MIT) is also central: it enables transformer-style self-attention models to do imputation effectively and benefits RNN models like BRITS when added. In benchmarking, SAITS reduced mean absolute error by 12% to 38% compared to BRITS, while training 2.0 to 2.6 times faster. Compared to a Transformer trained under the same joint-optimization strategy, SAITS achieved 2% to 19% lower MAE on the studied benchmarks, including 11% lower MAE on the Electricity dataset, while using only 15% to 30% of the Transformer's parameters. In a downstream mortality-prediction experiment, classifiers trained on SAITS-imputed PhysioNet data outperformed those trained on BRITS-imputed data by 1.3% ROC-AUC, 1.9% PR-AUC, and 1.4% F1, and outperformed Transformer-imputed data by 1.8% PR-AUC and 1.5% F1.

Key points

SAITS's core module is diagonally-masked self-attention (DMSA); an ablation confirmed that diagonal masking improves imputation.
The full model links three DMSA blocks in a cascade: blocks 1 and 2 are fused by a weighted combination into a learned representation, which feeds block 3; another weighted combination gives the final imputation.
A joint-optimization training strategy (with imputation task) lets self-attention models perform time-series imputation and also helps RNN models such as BRITS.
Across benchmarks, SAITS reduced MAE by 12% to 38% versus BRITS and by 2% to 19% versus Transformer under the same joint optimization.
SAITS achieved an 11% MAE reduction over Transformer on Electricity and used only 15% to 30% of Transformer parameters.
Downstream classification on SAITS-imputed data beat BRITS-imputed data on ROC-AUC/PR-AUC/F1 by 1.3/1.9/1.4 percentage points and Transformer-imputed data on PR-AUC/F1 by 1.8/1.5 points.

What is the architecture of SAITS and how does it improve imputation accuracy compared to BRITS and Transformer?

Related questions