Presenter
Bartłomiej Włodarczyk (@bartekwlodarczyk)
Abstract
Large collections of digital resources lack descriptive metadata due to the time constraints and understaffing in cultural heritage institutions, which make it difficult to assign such metadata.
To examine the potential of open models for metadata generation and enrichment, our presentation focuses on the performance of selected LLMs in extracting topics and themes from 48 Polish free-text responses to a survey conducted by the metadata and AI task group of the DCMI Education Committee. The survey concerned the use of AI in metadata tasks, with participants worldwide, including a significant share of responses from Poland. We selected seven open models, such as gpt-oss-safeguard:20b and deepseek-r1:32b, and ran using Ollama to evaluate their effectiveness in the extraction task. None of the models have been locally trained solely on Polish datasets.
The task involved the extraction of keywords corresponding to topics and themes. After each model completed the task, one of the authors evaluated all proposals and selected from 3-5 keywords per response. Then, standard information retrieval metrics such as precision, recall, and F1 score, were computed to evaluate the performance of each model. The results show that all models achieved moderate F1 scores for topic extraction (from 0.36 for deepseek to 0.56 for gpt-oss-safeguard) but struggled with theme extraction (from 0.18 for qwen3 to 0.36 for magistral). The comparison with the outcomes of the study concerning a similar task for Chinese responses and using three smaller open models shows a similar overall trend, although with lower values, especially for topics.
The experiment demonstrates that open models extract direct topics better than more general themes. The resulting topics could be connected to the existing linked-data vocabularies or domain-specific controlled vocabularies for better information retrieval.