An efficient late-interaction ModernBERT architecture for interpretable token-level semantic matching in Turkish
Journal of Engineering Research (Kuwait), 2026 (SCI-Expanded, Scopus)
- Yayın Türü: Makale / Tam Makale
- Basım Tarihi: 2026
- Doi Numarası: 10.1016/j.jer.2026.08.006
- Dergi Adı: Journal of Engineering Research (Kuwait)
- Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Scopus, Arab World Research Source, Directory of Open Access Journals, Academic Search Ultimate (EBSCO)
- Anahtar Kelimeler: Late-interaction architectures, LLM-assisted data annotation, Morphologically rich languages, Neural information retrieval, Token-level relevance classification, Turkish natural language processing
- Marmara Üniversitesi Adresli: Evet
Özet
Developing models that balance contextual understanding and computational efficiency remains a challenge in natural language processing, particularly for morphologically rich languages such as Turkish. This study proposes a hybrid Late-Interaction Fusion model based on ModernBERT to improve token-level semantic matching in Turkish texts. The model preserves the scalability of Bi-Encoders while incorporating contextual depth through a Multi-Head Attention-based late-interaction mechanism. The novelty of this work is that, unlike ColBERT-style late-interaction retrievers which combine pre-computed representations through a parameter-free similarity operator, the proposed fusion layer is a learned, query-conditioned cross-attention block that emits a calibrated relevance probability for every individual token, and that it is instantiated on a Turkish ModernBERT backbone trained on a two-million-pair corpus built through evidence-grounded LLM-assisted annotation. Experimental results show that the model attains a token-level F1-score of 0.557, closely approaching the span-based GLiNER baseline (0.580) while requiring substantially less computation, and clearly outperforming lexical matching (0.065), dense retrieval (0.129) and a standard Bi-Encoder (0.481). Additionally, the shared encoder enables fast convergence and the highest Macro-F1 (0.597) among the compared Turkish BERT variants in a downstream classification task, even with limited training data.