IMAGES-AI: Generative AI to improve products for the US Navy warfighter

Open Access
Article
Conference Proceedings
Authors: Bryan Croft, Seana Rothman, Tai Nguyen
Abstract

Project IMAGES-AI uses generative AI (GenAI) to improve products for the US Navy warfighter. Recent advances in GenAI, and the ability to adapt it to a specific domain, have made it a useful tool for enhancing the interfaces that display information used by the warfighter. The overarching goal is to improve situational awareness and strengthen the decision-making process, primarily through visual GenAI image generation. GenAI image generation has become a fast-growing capability and has overcome many of the early errors and mistakes seen in the technology's early days. Much of this has been made possible with improvements to Large Language Models (LLMs). LLMs can be very large and cover a wide range of data to support correct image generation. However, for very specific domains, much of the data required, such as for the US Navy warfighter and related topics, can be very limited, which can impact the results of GenAI-generated imagery for this specific topic. Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique that is used to customize pre-trained models (LLMs). It does this by training a small set of extra weights instead of altering the massive LLM model. The concept is to freeze the base weights of the original pre-trained model to preserve its coverage and capability. A domain-specific lightweight companion matrix or adapter layers can be added to inject domain-specific data into specific parts of the model with attention mechanisms. No retraining of the LLM is required; instead, a smaller add-on portion is used, where gradient updates modify only the much smaller set of adapter parameters. This produces a LoRA file, essentially a plug-in that adjusts the final output to better match the desired characteristics. In this case, applicable to US Navy applications. This technique is cost- and resource-efficient, small in file size compared to LLM data, and easy to swap out for different adapters to align with the desired output. IMAGES-AI has leveraged this technique to generate imagery that better illustrates Navy missions and courses of action. IMAGES-AI has been applied to applications to test and verify whether improvements are seen in warfighters through improved situational awareness and rapid decision-making. Domain and environmental information become embedded in the imagery. This has been examined through integration into applications and used in storyboarding workflows and scenario development. Initial prototype results indicate a valid return on investment.

Keywords: Generative AI, LoRAs, LLMs, Course of Action, Mission Planning, Image Generation, Storyboarding, ComfyUI, Shot List, Workflow

DOI: 10.54941/ahfe1008407

Cite this paper
Downloads
0
Visits
1
Download PDF

More from this volume

← Exposing Tool Use in AI-Assistant Interfaces: Inspectable Evidence for Industrial AssistanceApplying Formal Methods to an Augmented Reality Interface →
View all articles in Human Factors in Design, Engineering, and Computing →