Artificial intelligence has long been a cornerstone of cybersecurity operations. From malware detection to network traffic analysis, predictive machine learning models and other narrow AI applications have been used in cybersecurity for decades.¹ Yet recent advances in general-purpose AI, along with advances in predictive modeling, have ushered in a new generation of defensive applications and national security use cases.² Threat-actors are already making use of these new tools.³ Defenders can now begin to use frontier AI systems to identify vulnerabilities across codebases and to understand, in plain language, their overall security posture. They can also use those systems to quickly summarize incidents, facilitate rapid incident response, and perform a wide array of other tasks that are foundational to modern cybersecurity best practices.⁴

As part of our publication series on AI security, this issue brief discusses several key ways in which frontier AI may improve cyber defense. Given that advanced general-purpose AI systems are inherently dual-use, identifying, developing and deploying their defensive capabilities before their malicious applications arise is essential. We hope this brief will serve as a useful starting point for cybersecurity professionals considering how to leverage frontier AI systems for defensive purposes.  

ADVANCED AI USE CASE EXAMPLES IN CYBER OPERATIONS AND SECURITY IMPROVEMENTS

Large language models can be beneficial in several areas, including process automation, querying and analysis, vulnerability discovery, open source intelligence gathering, and training. 

Process Automation. Defenders who leverage advanced AI for process automation can increase the speed, scale, and accuracy with which they respond to new incidents and attacks.⁵ Cybersecurity tasks and processes that can benefit from automation include:  

• Alert triage. AI can help to assess and prioritize security alerts based on severity and context, ensuring that the most critical issues receive immediate attention. AI can also correlate alerts and events that might not otherwise be connected by human responders. 
• Alert routing. AI-assisted systems can help to ensure that alerts are directed to the most appropriate incident responders, improving response times and efficiency.
• Incident response. Agents can be integrated into incident tracking and response flows for automated ticket enrichment, testing hypotheses against data sources involved in incidents to gather additional information and provide an action recommendation quickly and efficiently.
• Identity and Access Management (IAM). AI can help to streamline IAM processes while maintaining security principles like least privilege, including by generating intelligent access reduction recommendations and policies based on user roles and contextual organizational factors gathered from sources across an enterprise, such as employee roles.

Significantly, important decisions would still need to be made by a human in each case above; alerts would still go to a security team, while IAM decisions would be verified before implementation. However, process automation can enable those decisions to be made more rapidly, more accurately, and more efficiently.  

Querying and Analysis. Frontier AI may enable cybersecurity professionals to query and receive responses about incidents and security posture in natural language. Key capabilities include:

• Forensic analysis. AI can support the analysis of forensic evidence, including timeline events, live response data, and memory dumps, and cross-referencing each with system and activity logs. 
• Pattern identification. AI and RAG-based systems can summarize and identify patterns from large volumes of logs and alerts, even as systems constantly upgrade and change, without requiring new programming. 
• Threat remediation advice. AI can recommend next steps for threat remediation by quickly synthesizing information from across an organization and leveraging a semantic understanding, generated from system diagrams, about how the organization’s systems are connected and how threats may move across them.  
• Incident analysis. When a security incident occurs, AI can assist responders by rapidly analyzing the situation and providing actionable mitigation recommendations.
• Integrated translation. Since frontier AI can understand many of the most spoken languages and threat actors operate globally, it can allow for the translation and contextualization of tactics, techniques, and procedures (TTPs) into broader threat analysis.
• Comprehensive threat reporting. AI systems can analyze vast amounts of security data to help analysts generate detailed threat reports, such as summaries of recent security incidents, emerging threat trends, and organization-specific vulnerabilities.
• Data integration. AI can combine and analyze data from multiple sources. For example, it might synthesize firewall logs, intrusion detection system alerts, behavioral signals, and external threat feeds to create a daily threat landscape report.
• Binary analysis. With access to existing tools such as decompilers and disassemblers, AI can uplift defenders by helping them to analyze the output of those tools and provide a better understanding of the behavior of malware. 

Vulnerability Discovery and Patching. Frontier AI can also improve vulnerability discovery and remediation. Thanks to a variety of factors—from greater use of coding examples in post-training to ever larger context windows—advanced AI systems are increasingly useful for identifying and mitigating vulnerabilities.⁶ Key applications include:

• Software fuzzing. AI can be used as a tool for automated software testing, writing test harnesses that generate invalid, unexpected, or random data as input to identify crashes, memory leaks, or potential vulnerabilities. When a corruption is found, AI can write a proof-of-concept that confirms or rejects the finding. AI can expand code coverage and triage the findings by severity.⁷ 
• Automated penetration testing: Frontier AI systems are now capable of conducting aspects of penetration testing that previously required extensive human expertise. For example, PENTESTGPT demonstrates how LLMs can effectively use testing tools, interpret outputs, and propose subsequent actions in penetration testing workflows.⁸
• Edge case detection. AI systems tailored for coding and software tasks can systematically generate and test various inputs, uncovering a greater number of edge cases and unexpected behaviors that could lead to security issues.
• Automated fixes. AI systems capable of not only identifying but also fixing software vulnerabilities at scale are currently under active research, including initiatives such as the DARPA AI Cyber Challenge.⁹ Continuing the trend of “shift left” on security, AI is increasingly used to identify insecure coding practices in Continuous Integration systems and automatically suggest fixes.

While advanced AI systems are useful for vulnerability detection and patching today, agentic systems that use reasoning, planning, and trial-and-error approaches similar to human cybersecurity practitioners are actively under research and development.¹⁰ Such systems will likely further improve defenders’ ability to identify and address potential weaknesses before they can be exploited by malicious actors.

Open-source intelligence gathering and reporting. Frontier AI can significantly improve open source intelligence (OSINT) gathering and analysis. Key applications include:

• Real-time monitoring and analysis. AI can continuously monitor online forums, social media platforms, and dark web sources for new or evolving threats, providing security teams with timely updates and threat actor TTPs.
• Intelligence report generation. AI can synthesize the gathered information into comprehensive OSINT reports for sharing between entities.
• Trend identification. AI can identify emerging trends and patterns in online discussions and activities related to cybersecurity threats, providing an early warning system for shifting threats against sectors and technology used by an organization.

These capabilities can help cybersecurity teams stay ahead of potential threats by providing them with comprehensive, timely, and actionable intelligence gathered from diverse and widely-available open sources.

Exercises, simulation and training. Frontier AI can also improve cybersecurity training and preparedness. Key applications include:

• Tabletop exercise generation. AI can help to create diverse and complex scenarios for cybersecurity tabletop exercises, enhancing team preparedness.
• Realistic phishing simulations. AI can help to generate convincing phishing attempts tailored to an organization’s specific context, improving staff preparedness.
• Personalized training programs. AI can help to tailor cybersecurity training to an individual’s skill level, role, and learning pace.
• Adaptive scenarios. AI can help to create and adjust training scenarios in real-time based on a trainee’s responses and performance.

By helping to provide more effective, relevant, and up-to-date cybersecurity training, frontier AI can help defenders to improve their overall security posture and readiness to face evolving threats.

LIMITATIONS

While advanced AI systems offer significant potential for cybersecurity, they also have important limitations.¹¹

• False Positives and Negatives: AI can generate false positives and negatives, leading to missed threats or unnecessary actions. Proper model training and calibration is essential to reduce these issues along with the implementation of human oversight for the most critical workloads.
• Adversarial Attacks: AI systems may be manipulated by adversaries in unique and evolving ways, exploiting model weaknesses. Understanding and addressing adversarial threats is critical to maintaining trust in AI defenses as the field advances.
• Explainability and Transparency: AI models often act as “black boxes” to users, making it difficult to explain decisions. Lack of transparency can hinder trust and adoption in critical operations, requiring clear performance and risk evaluations. Tooling should cite sources by URL when appropriate.
• Data Quality and Privacy: AI systems require high-quality training data to perform well, but gathering this data while protecting privacy presents a major challenge. While promising privacy-preserving technologies are emerging from research, practical implementation remains a key hurdle.
• Scalability: Significant computational resources are often needed for the most advanced general-purpose AI systems. For organizations with resource constraints, smaller distilled models, or a combination of models, may be used to make up for this lack of computational resources but may come with accuracy tradeoffs.
• Bias: AI models may inherit biases from training data, potentially overlooking emerging threats. Regular evaluation and diverse data sets are needed to mitigate bias.
• Ethical and Regulatory Compliance: Ethical use and adherence to evolving regulations are essential.  Evolving regulatory and safety considerations will affect model availability and applicability.

CONCLUSION

As the capabilities of general-purpose AI continue to grow, so too will its potential uses for cybersecurity. As noted above, however, leveraging that promise does not mean automating as many systems as possible, but instead designing those systems with a strong focus on human-AI collaboration. 

AI systems work best when they are designed from the start with the goal of assisting human operators rather than replacing them, and with appropriate guardrails in mind. This can be accomplished in part by implementing mechanisms for an AI system to request human input on critical decisions and incorporating feedback from security professionals to refine system outputs. Models can be trained and enhanced based on how humans handle their initial recommendations. In some cases, full automation may be possible, but should always include a mechanism for human oversight of automated decisions. It is also important to recognize that AI is not suitable for all cybersecurity tasks, such as those requiring perfect accuracy.

With the appropriate safeguards and frameworks in place, frontier AI systems have the potential to help uplift cyber defense. As the threat landscape changes and adversaries begin to use AI in their offensive operations, it will become increasingly important that defenders stay one step ahead. Advanced general-purpose models can be a powerful tool for cybersecurity practitioners to enhance their existing cyber defenses and fortify their overall cybersecurity posture. 

Footnotes

1. For more on the different types of machine learning algorithms used and their applications, see “Artificial Intelligence and Cybersecurity Research” (ENISA, July 2023), p. 10. ↩︎
2. See Sarah Mercer and Tim Watson, “Generative AI in Cybersecurity: Assessing impact on current and future malicious software” (Alan Turing Institute CETaS Briefing Papers, June 2024), p. 20-22.  ↩︎
3. For example, a North Korean threat actor has used large language models to assist in vulnerability research and spearphishing campaigns; a Russian intelligence unit has used them to inform reconnaissance and scripting techniques; and an Iranian threat actor has used them to generate long-form articles and social media posts. See “Staying ahead of threat actors in the age of AI” (Microsoft Threat Intelligence, February 14, 2024) and “Disrupting a covert Iranian influence operation” (OpenAI, August 16, 2024).  ↩︎
4. For a good overview of current and future uses of AI in cybersecurity, see Ramanpreet Kaur, Dusan Gabrijelcic, and Tomaz Klobucar, “Artificial Intelligence for cybersecurity: Literature review and future research directions” (September 2023). ↩︎
5. In line with early findings on generative AI and worker productivity, such as Erik Brynjolfsson, Danielle Li and Lindsey Raymond’s NBER working paper “Generative AI at Work” (2023), a recent randomized control trial by several researchers at Microsoft found a meaningful reduction in security incident mean time to resolution among security operations center staff with access to general purpose AI systems. See James Bono, Justin Grana and Alec Xu, “Generative AI and Security Operations Center Productivity: Evidence from Live Operations” (working paper, 2024). ↩︎
6. Most notably, a security research team at Google was able to use an AI agent to find an undiscovered zero-day vulnerability in the popular SQLite Database Engine. See Big Sleep Team, “From Naptime to Big Sleep: Using Large Language Models To Catch Vulnerabilities In Real-World Code” (November 2024). ↩︎
7. For example, Fuzz4All is an open-source fuzz testing framework that provides a unified platform for running fuzzing campaigns across multiple programs and environments.The LLM integration helps optimize fuzzing campaigns by providing insights, generating effective test cases, and assisting in result interpretation, ultimately improving the overall effectiveness and efficiency of security testing. ↩︎
8. Such frameworks can significantly increase testing coverage and efficiency, while still working alongside human security professionals who provide oversight and handle complex decision-making. See Gelei Deng et al, “PENTESTGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing” (working paper, 2024). ↩︎
9. The Artificial Intelligence Cyber Challenge (AIxCC) is a two-year competition, established by DARPA and ARPA-H, that brings together competitors from various AI research and development institutes and aims to advance our understanding of how to develop AI systems capable of securing critical code. ↩︎
10. Some frontier AI systems have shown progress in reasoning capabilities in the cybersecurity domain. For example, see the OpenAI o1 System Card, p. 15-16, which documents the attempts of the pre-mitigation o1 model to solve a CTF challenge. ↩︎
11. See Jessica Ji, Jenny Jun, Maggie Wu, and Rebecca Gelles, “Cybersecurity Risks of AI-Generated Code” (Georgetown University Center for Security and Emerging Technology, November 2024). ↩︎