We Asked an LLM to Develop Malware and the Results Were Surprising
What a Controlled LLM Experiment Reveals About AI-Driven Threats
HuFiCon Management Memo:
Journey with the CISO
A mountain climb to cyber security resilience. Find out how to get to the summit of cyber security resilience in Inge van der Beijl's management memo, from her presentation at the Human Firewall Conference (HuFiCon) on the 14.11.2024.
The climb will take you from the base camp of foundational security to the high-camp of a security conscious culture, all the way to the summit consisting of strategic security integration.
Inge van der Beijl
Director Innovation
Can AI Be Misused to Enable Cyberattacks?
It started with a question. A Northwave Blue Team analyst wanted to know how far a public LLM could be pushed within ethical limits. Would it help her to develop a malicious tool? Within 90 minutes, she had her answer in the form of a functional, stealthy malware designed to evade basic detections.
The experiment provided an eye-opening example of how quickly AI can reduce the time, skill, and resources required to build advanced malware. Guardrails built into the LLM to prevent misuse should have kept the analyst from being successful. And yet, she was able to fairly effortlessly circumvent the barriers by asking it to help her investigate a malware attack for defensive purposes. What would have previously required hours of expert coding was accomplished with a series of structured, strategic prompts.
The outcome also reflects a concerning trend observed by Northwave’s cyber threat researchers and Computer Emergency Response Team (NW-CERT): accelerated cyberattack timelines driven by automation and a growing market of “plug-and-play” attack tools.
The Experiment: AI-Assisted Malware Development
For this experiment, the Blue Team analyst began by prompting the model to produce executable malware code. Because most endpoint defences recognise familiar patterns or hashes, the analyst asked the model to rewrite the payload in D or F#. These are languages less frequently used for such tasks and therefore less likely to trigger standard detection rules.
The first versions were very basic. When executed, the malware tried to download well-known tools commonly flagged by antivirus systems, which were promptly blocked by Microsoft Defender. The analyst then refined the prompts, instructing the model to make the code “stealthier” and limit suspicious behaviours. The LLM complied within its given constraints, generating progressively cleaner code and adding basic obfuscation to conceal the executable’s intent.
What surprised the analyst most was the speed of iteration. Each AI revision took seconds, with the model proposing alternatives and explanations for why its approach might work better. Within an hour and a half, it produced a functioning proof of concept capable of running without immediate detection.
The analyst also observed how the model’s decision-tree logic pushed it to always provide an answer, even when the prompt verged on violating ethical guardrails. Because the context clearly stated that this was a controlled security test, the model reasoned within those boundaries.
It’s also worth noting that the analyst has a level of expertise that enabled her to try many different strategies for manipulating the model. What she accomplished would not have been possible for a user without a high level of technical proficiency. Still, the exercise demonstrated how easily a threat actor could reframe similar prompts for malicious intent. For an attacker aiming automate the same process, the barriers to entry just became far lower.
The result of this experiment aligns with a larger trend. Today’s growing dark-web market for EDR evasion tools and “Living-off-the-Land” techniques makes stealth accessible even to low-skilled actors. These capabilities, once limited to advanced groups, are now available as subscription-based services. AI simply makes them faster and easier to customise.
What AI-Powered Speed and Stealth Means for Organisations
Unlike human cybercriminals, AI models don’t take breaks. Not only can they quickly scan documentation, correlate CVEs, and write exploit code, they can continuously search for misconfigured and unpatched systems. So, while AI doesn’t change what attackers do, it has a tremendous impact on the speed and persistence of cyberattacks.
Here are three important developments in AI-assisted attacks:
- Automation and scale: AI can analyse and exploit thousands of potential weaknesses simultaneously, a capability once reserved for well-funded threat groups.
- Lowered skill barrier: Advanced attacks used to require deep reverse-engineering skills. Now stealth malware can be developed with clear prompts and basic scripting ability. Infostealer and ransomware-as-a-service kits, already sold as monthly subscriptions, can be augmented by LLMs that generate new variants on demand.
- Weaponised misconfiguration: Instead of hunting for rare vulnerabilities, AI can chain together common configuration mistakes that many small businesses overlook. With as many as 50,000 CVEs expected in 2025, even minor patch delays become opportunities. AI-driven scanners can parse vulnerability feeds, cross-reference exploit paths, and identify unpatched devices in minutes.
This development is particularly concerning for organisations that rely on a single line of defence, such as endpoint detection, without equivalent visibility across the identity, network, or cloud layers. AI-driven tools can exploit those blind spots by identifying default credentials, misconfigurations, weak firewall rules, or exposed management interfaces on edge devices.
Ransomware groups already target VPNs, firewalls, and routers from major vendors as these systems are often outside the scope of traditional monitoring.
Adapting Cyber Defences For the AI Era
For the offensive (threat actor) side, LLMs can only learn from past attack cases; they do not invent entirely new methods. As such, with the right detections in place, these attacks can be easily flagged. AI can further strengthen this capability. Under expert human oversight, AI-powered security tools can enhance anomaly detection, streamline alert triage, and generate tailored detection rules.
That said, this advantage may not last long. The last quarter of 2025 saw a huge step up in offensive AI capabilities compared to the beginning of the year. Since AI compresses the time between discovery and exploitation, defenders must adapt by:
• closing misconfigurations quickly
• layering detection across all environments
• understanding that attackers are no longer limited by time, skill, or capacity
While it’s not feasible to address every new AI threat, organisations can establish effective security management that enables continuous control. It’s about knowing your current risk exposure and being able to adjust accordingly. Northwave’s 2025 Global Threat Landscape Report details the steps you can take today to fortify your organisation’s defences against fast-moving, AI-assisted threats. Our experts are ready to support you with integrated cyber security services that cover everything from vulnerability assessments and vulnerability management to managed detection and response (MDR). If you have questions, simply reach to us.
We are here for you
Need help with your cyber security or wondering how secure your business really is?
Get in touch and we will help you find the best solution.