To boldly go where AI must not go alone: Designing for non-delegable human authority in AI-assisted expert work

Abstract

AI-assisted expert work raises a recurring question for human-machine systems design: where must human authority remain structurally located, rather than delegated to behavioural monitoring of the automated system? Conventional oversight is operationalised as post-hoc error detection, which the human factors literature shows fails under complacency and vigilance decrement at high reliability. We propose a design pattern that re-frames oversight as a structural property of the design process rather than a behavioural demand on the operator. We define Non-Delegable Points (NDPs): procedural stages whose constitutive quality criteria are human-authored and which resist delegation regardless of agent capability. For qualitative content analysis, we identify three NDPs – domain-knowledge curation, iterative codebook refinement, and communicative validation – and present an architecture enforcing them through codified process steps, immutable knowledge layers, and strict separation of a development agent from an application agent. The architecture suggests reproducible per-step audit artefacts without depending on reasoning-chain faithfulness. We anchor the pattern in the automation-monitoring tradition and link it to the EU AI Act's requirement for effective human oversight (Article 14). The contribution offers a transferable model for AI-assisted expert work in which human authority is enforced by design.

Keywords: Human-Machine Systems, Design Process, Models And Approaches, Non-Delegable Points, Human Oversight, Audit Trail

DOI: 10.54941/ahfe1008078