Jinbiao Yang

List of John Benjamins publications in which Jinbiao Yang is involved.

Article

Yang, Jinbiao 2024 Rethinking tokenization: Crafting better tokenizers for large language models The Unit of Processing in Chinese, Wang, Tianlin (ed.), pp. 94–109 | Article

Tokenization significantly influences language models (LMs)’ performance. This paper traces the evolution of tokenizers from word-level to subword-level, analyzing how they balance tokens and types to enhance model adaptability while controlling complexity. Despite subword tokenizers like Byte… read more