Shaping Conversations: Custom GPTs to Spark Reflection in Design

Abstract

This paper documents the development and evaluation of a customized GPT designed to facilitate metacognitive reflection on cognitive biases in design decision-making. Leveraging the OpenAI GPTs platform, a specialized conversational agent was created by integrating a structured knowledge base of twelve categories of cognitive biases with prompt engineering strategies oriented toward progressive disclosure. The system aimed to stimulate reflective thinking in design students by gradually surfacing relevant biases through contextual interactions rather than through systematic enumeration.The paper details the construction of the chatbot, including the design of system instructions, the organization of the knowledge base, and the definition of conversation starters used to initiate dialogues. Particular emphasis was placed on turn-taking management and the principle of one-question-per-turn, in line with established theories of cognitive load management. Despite these design directives, the analysis of conversations conducted with a sample of design students revealed significant limitations in the effective implementation of conversational principles. In seven out of ten interactions, the system presented full lists of biases in sequence, contradicting the intended principle of progressive disclosure and generating cognitive overload.Empirical analysis highlights three main issues: structural rigidity, with responses derived from repetitive templates and limited adaptability to specific project contexts; information dumping, characterized by lengthy outputs and simultaneous multiple questions; and restricted personalization, reduced to superficial lexical substitutions without meaningful contextual selection of relevant biases. These shortcomings reflect the architectural constraints of the platform at the time of the study (early 2024), when persistent memory and advanced retrieval mechanisms for dynamic context management were not yet available.Although the outcomes diverged from the original objectives, the work offers valuable contributions. First, it provides a systematic documentation of the iterative development process of a customized GPT for design reflection, emphasizing the need to balance prescriptive directives with conversational flexibility in prompt engineering. Second, it identifies practical considerations for the design of educational chatbots: the necessity of more sophisticated architectural controls to prevent enumeration, the importance of conversational management policies (anti-repetition, recap, one lever per turn), and the role of knowledge bases in contextual adaptation. Finally, the paper discusses how technological advances introduced in late 2024, such as persistent memory in GPT models and the integration of retrieval-augmented generation (RAG), may mitigate some of the observed failure modes, opening pathways for more effective implementations in the future.In conclusion, this study contributes to the understanding of how conversational AI platforms can be tailored to support reflective processes in educational and design contexts. While the experiment revealed the limitations of prompt-engineering-based solutions, the findings underscore the potential of specialized GPTs as metacognitive support tools and as catalysts for embedding reflective practices in design education.

Keywords: Personalized GPTs, cognitive biases, prompt engineering, interaction design

DOI: 10.54941/ahfe1007066