Assessing the potential of using large language models for pragmatic annotation of historical texts

This study investigates the viability of using large language models (llms) to conduct pragmatic annotations of historical texts. The investigation employs a small corpus of witness depositions and compares Claude 3.5 Sonnet — an llm that excels in reasoning over text — with two human annotators over their performance in the pragmatic annotation of Early Modern English (emode) texts. The study also compares the model’s annotations on modernised and original versions of the corpus to explore if emode spelling variations affect its performance. The results revealed that although the model’s annotations were less satisfactory than human annotators’, it achieved moderate inter-coder agreement and balanced precision and recall, which is desirable in this particular task by maximising identification without sacrificing accuracy. Furthermore, the prevalent spelling variations did not significantly impair the model’s ability to recognise epistemic stance in the original emode texts. Therefore, we propose a human–ai collaboration approach for historical pragmatic annotation.