Enhancing Programming Task Performance with LLMs: The Role of Query Formulation and Task-Technology Fit

Abstract: This study investigates the effectiveness of large language models (LLMs) in solving programming tasks, with a particular focus on how query formulation influences response quality. Using the Task-Technology Fit (TTF) framework, the research explores the alignment between task requirements and LLM capabilities, and how this alignment impacts student performance.An experiment was conducted with 60 students from the Faculty of Informatics and the Faculty of Engineering at the Juraj Dobrila University of Pula. Participants were asked to solve the “8 Queens” problem in two 30-minute phases: first independently, and then with assistance from the ChatGPT-4 model, during which they could issue 5–10 iterative queries. This setup enabled a comparative analysis of student performance in traditional versus LLM-assisted conditions. Data collection included a structured questionnaire aligned with TTF constructs. The data were analyzed using Partial Least Squares Structural Equation Modeling (PLS-SEM), which modeled the relationships between task characteristics (TASK), technology characteristics (TECH), task-technology fit (TTF), and user performance (PERF). The results indicate that both task complexity and technology capabilities significantly contribute to TTF. In turn, TTF strongly influences user performance with the model explaining 47.3% of the variance in TTF and 33.2% in performance.Statistical analysis confirmed a significant improvement in solution accuracy when tasks were completed with LLM support. Furthermore, qualitative analysis showed that well-structured, context-rich queries led to more accurate and relevant model responses. The findings underscore the pivotal role of query formulation in optimizing the use of LLMs for programming tasks. Developing effective query strategies enhances task-technology alignment and ultimately improves performance. This study contributes to a growing understanding of human–AI interaction and highlights the importance of integrating query design skills into educational and professional programming contexts.

Keywords: large language models (LLMs), query formulation, Task-Technology Fit (TTF), programming performance, PLS-SEM, human–AI interaction

DOI: 10.54941/ahfe1006784

Cite this paper:

​