Evaluation of Gen-AI (LLM) Results on Prompts in the Batak Language

Abstract

The rapid advancement of Large Language Models (LLMs) has
significantly enhanced natural language processing capabilities;
however, their performance on low-resource regional languages
remains underexplored. The Toba Batak language, a regional
language spoken in North Sumatera, Indonesia, faces limited
representation in global AI training datasets, raising concerns about
linguistic accuracy and cultural preservation in AI-generated
content. This study evaluates the performance of four Large
Language Models—ChatGPT, Gemini, Perplexity, and Toba-LLM—
in generating responses to prompts written in the Toba Batak
language. The evaluation follows the CRISP-DM methodology and
employs a structured dataset consisting of categorized prompts and
corresponding ground truth answers. Model outputs are assessed
using six quantitative metrics: lexical similarity, keyword overlap,
length ratio, cosine similarity, cultural relevance, and fluency. The
results indicate that general-purpose models such as ChatGPT and
Gemini outperform the local Toba-LLM in terms of semantic
similarity and fluency, while Toba-LLM demonstrates higher lexical
alignment and response conciseness. However, cultural relevance
remains a challenge across all models, highlighting the difficulty of
capturing culturally grounded knowledge in low-resource language
settings. These findings underscore the need for culturally aware
evaluation metrics and enriched local language datasets to improve
the applicability of LLMs for regional language preservation and AI-
based educational applications.