Current Issue [Vol. 12, No. 01] [January 2026]

Transformer models have significantly advanced Natural Language Processing (NLP) by replacing recurrence with self-attention mechanisms [1]. While standard Transformers compute attention uniformly across heads and tokens, this may dilute task-specific importance in classification problems [2,8]. This paper proposes a Weighted Self-Attention Optimization (WSAO) framework that introduces adaptive token-level weighting into the Transformer encoder to enhance discriminative feature learning. Using a public benchmark dataset, we demonstrate that the proposed formulation improves classification performance compared to baseline Transformer [1] and LSTM models [3]. Mathematical formulation, experimental evaluation, and comparative analysis are presented to highlight the effectiveness of the approach.
Keywords: Transformer Models, Self-Attention Optimization, Text Classification, Cross-Entropy Loss, NLP