USING LLM IN DISTANCE LEARNING: ADAPTING ENGLISH-LANGUAGE ALGEBRAIC TASKS

Abstract

The COVID-19 pandemic and the full-scale aggression against Ukraine have underscored the importance of remote learning. Consequently, the demands on both the quality of software systems and the educators who prepare lecture courses and knowledge assessments have increased. The market has already seen the emergence of many free and commercial solutions based on large language models (LLMs) such as ChatGPT, Claude, Google Gemini, etc., which are already being used by students to prepare their assignments. The integration of LLM/AI into the educational process, particularly in distance education, where the variety of tasks, their explanations, and interpretations are crucial, presents a new challenge that requires thoughtful consideration and practical implementation. One potential approach is the adaptation of well-known English-language task sets into domestic distance learning systems for Ukrainian-speaking students. However, this process will only be meaningful if the AI is capable of generating translations (adaptations) of sufficient quality to meet educational requirements. Addi-tionally, students are already attempting to use AI as an assistant in solving tasks in exact sciences, but the accuracy of AI solutions for tasks in the Ukrainian language remains underexplored. The aim of this article is to analyze the current possibilities for distance education in the context of the active proliferation of LLM-based systems regarding the adaptation of English-language school-level algebra tasks. The objectives of the article are as follows: 1) to analyze the ability of two popular LLM models to generate translations (adaptations) of algebraic tasks according to specified rules; 2) to analyze the ability of LLM models to solve tasks in their original form; 3) to analyze the ability of LLM models to solve tasks in their adapted form; 4) to analyze the ability of LLM to explain the solution process of tasks in detail in Ukrainian. The primary source for verification was the GSM8K task set (https://github.com/openai/grade-school-math, a collection of school-level math problems created by a professional teachers). Subsequently, ChatGPT v4 / v4 Omni and Gemini 1.5 PRO Preview adapted 200 tasks from this set into Ukrainian, replacing English names with Ukrainian ones and converting units of measurement (without actual recalculation, simply for convenience and understanding by students, e.g., "gallon" became "liter" and, US dollar — hryvnia, etc.) To assess the quality of the Ukrainian adaptation, the Open Source software, called LanguageTool (https://languagetool.org/) was used. Additionally, a human conducted a visual check for the correctness of the adaptation of selected questions through random sampling from the translated data set. The adapted tasks had to be understood not only in English but also in Ukrainian and solved by the LLM models. The results were also compared with the reference answers from the set. The findings were as follows: 1) both LLM models demonstrated almost identical grammatical quality of the Ukrainian translation — about 70%. 2) LLMs demonstrated solution accuracy of the original tasks at the level of 85% (ChatGPT v4), 93.5% (ChatGPT v4 Omni), and 75% (Gemini) respectively. There is a clear dependence not only on the structure of the prompt for the system, but also on the volume of output and the level of detail required by the system. If the LLM performs several different tasks (translation/solution) within one request, the results of mathematical calculations will be much worse than when the LLM is only required to show the detailed path to the final result. ChatGPT on the English variant showed 93.5% (85%) of solved tasks, which correlates well with the results of other studies in this field; 3) for the Ukrainian variant (ChatGPT adaptation), the accuracy differed by 5% from the original formulation (after analyzing typical errors and correcting the adaptation). For Gemini, the English variant showed ~25%, while the Ukrainian (own adaptation and ChatGPT adaptation) showed ~50% errors; 4) both systems demonstrated the ability to explain the solution path of the task, but require external control of the results. The development of LLMs shows enormous potential for this technology in distance learning systems in the Ukrainian language. Significant time savings in preparing numerous uniform tasks from English sources. AI demonstrates the ability to analyze tasks, and the quality of the solution does not significantly depend on the language of the original task. Right now, human control over the results of calculations proposed by AI is still required and should be considered mandatory.