What is the purpose of the context token in the Transformer encoder of TS-TCC, and how is it used in the Contextual Contrasting module?

The context token is a special token prepended to the encoded input sequence so that after passing through the Transformer layers, its final representation can be extracted as the context vector c_t. This context vector provides a summary of the historical context for the temporal contrasting prediction task. In the Contextual Contrasting module, c_t is projected by a non-linear head and used in a contrastive loss, where its positive pair is the context vector from the other augmented view of the same sample and negative pairs come from other samples in the batch.

In TS-TCC's Transformer encoder, the input representation z≤t is linearly projected and a special context token c is prepended, forming the initial sequence ψ0 = [c; z̃]. The sequence is processed through stacked Multi-Head Attention and MLP layers. From the final layer, the output corresponding to the context token is taken as the context vector ct. This vector represents the model's aggregated historical context up to time t, which the temporal contrasting module uses to predict future representations. The same context vector is also passed to the Contextual Contrasting module, which applies a non-linear projection head and then performs contrastive learning on the projected vectors. For a batch of N inputs with strong and weak augmentations, the 2N projected context vectors are compared, and for each context vector the positive sample is the context vector from the other augmented view of the same input. The loss maximizes cosine similarity between each context vector and its positive while minimizing similarity with the 2N−2 negative context vectors from other samples.