On the Lack of Phishing Misuse Prevention in Public Artificial Intelligence Tools

Abstract: Phishing remains one of the most common and effective forms of social engineering, with cybercriminals constantly refining their tactics to exploit human vulnerabilities. The sheer volume of phishing attacks is staggering: almost 1.2% of all emails sent are malicious. This equates to around 3.4 billion phishing emails per day. The effectiveness of phishing attacks is also underlined by numerous studies. Phishing is identified as the leading initial attack vector, responsible for 41% of security incidents. This means that practically every company is threatened by phishing attacks.In parallel, there have been rapid advances in the field of artificial intelligence (AI) in recent years, giving the general public access to powerful tools that can handle complex tasks with ease. However, alongside these benefits, the potential for abuse has also become a major concern. The integration of AI into social engineering attacks has significantly increased the opportunities for cybercriminals. Research has shown that AI-generated phishing emails are difficult for humans to distinguish from real messages. According to one study, phishing emails written by AI were opened by 78% of recipients, with 21% clicking on malicious content such as links or attachments. Although the click-through rate is still lower compared to human-crafted emails, generative AI tools (GenAI) can help cybercriminals compose phishing emails at least 40% faster, which can lead to a significant increase in phishing success rates. The increasing potential to use public AI tools for abusive purposes has also been recognized by the developers of AI models. Thus, publicly available AI tools often have built-in mechanisms to detect and prevent misuse. This paper examines the potential for misuse of publicly available AI in the context of phishing attacks, focusing on the content generation phase. In particular, the study examines the effectiveness of existing abuse prevention mechanisms implemented by AI platforms like fine-tuning, filters, rejection sampling, system prompts and dataset filtering. To this end, it is explored how prompts to the AI need to be altered for circumventing the misuse preventing mechanisms. While in some cases the simple request to write a phishing email succeeds, other AI tools implement more sophisticated mechanisms. In the end, however, all prevention safeguards could be circumvented. The findings highlight the significant threat posed by AI-powered social engineering attacks and emphasize the urgent need for robust defense in depth strategies against phishing attacks and increased awareness to mitigate the risks in the evolving digital landscape.In addition, the paper demonstrates that the quality of the AI tool varies in terms of the phishing emails generated. To this end, the phishing emails generated by circumventing the protection mechanisms of the AI are (subjectively) compared and evaluated by the authors. The preliminary conclusion is that automatically generated phishing emails of some public AI tools can certainly match that of manually generated emails. While the objective confirmation of this hypothesis requires further study even the subjective quality of the generated phishing emails shows the dimension of the problem.

Keywords: Artificial Intelligence, Social Engineering, Phishing, Spear Phishing, Misuse Prevention

DOI: 10.54941/ahfe1005893

Cite this paper:

​