AI-powered Cyberattacks
What could cyberattacks look like with advanced generative AI capabilities?
Springing from recent advancements in Large Language Models (LLMs), AI has been compared to nuclear weapons regarding the risks and potential damage it could cause to the world. While these risks vary from a malicious superintelligence that destroys humanity to wide-scale job displacement, a notable short-term danger has been their use in augmenting cyberattacks.
We’ve seen quite a bit of media coverage on this, but from a bit of Googling, it’s been unclear how exactly malicious actors are/could be leveraging AI for their campaigns beyond the basics of typing “write me malware code” or “write me a phishing email” into a ChatGPT-type interface (see WormGPT, FraudGPT). There’s no doubt harm caused by these applications, but when it comes to more sophisticated actors, it seems unlikely this is the only way they will use AI and that these are more script-kiddie tools.
This article looks at potential tools that could exist in the near future for “sophisticated” cyberattacks using AI. The goal is to enumerate these potential scenarios before these tools exist and these attacks take place to aid the construction of potential countermeasures.
The Softmax Syndicate
Below we look at the “Softmax Syndicate”, a fictional group of Ex-Machine Learning engineers who have turned to a life of cybercrime after losing their GenAI startup jobs during the great 2024 tech recession.
For these examples, we’ll assume they have amble access to GPU-compute and that for AI applications that run on a victim’s systems, they either use remote calls to their servers or run lighter-weight models locally. The hacking tools described are meant to be plausible in the near-ish future but may not be trivial to build given today’s state-of-the-art. Some of these techniques may also be possible without AI but are used to show how AI could be used.
In several places I’ll refer to the models as “agents”, by this, I mean a set of prompts and plugins that allow the LLM to iteratively take inputs and perform actions (learn more).
Deep Penetration Testing
For easier targets, the syndicate uses an LLM-based search tool to identify commonly known vulnerabilities in a target’s system. Given a set of websites or internet-facing hosts, ChatGPT-like agents scan system versions, ports, etc. to determine vulnerable surface areas and exploits.
This is largely an extension of what’s already been done by criminals and in whitehat penetration tests, the difference being the ability of an LLM-based agent to narrow the search space for potential exploits and its ability to generate novel penetration attempts.
Traditional pentesting (oversimplified):
Scan ports (ex. port 80)
Identify common services running on these ports (HTTP server)
Test pre-existing exploits for access (list of known XSS payloads)
Deep pentesting:
Scan ports and synthesize traffic for gathering information (ex. gather and parse webpage information)
Identify and synthesize context-specific exploits (craft XSS payloads specific to the software versions or combination of software identified from scraping and target extracted endpoints)
Test and adapt exploits based on responses (run the XSS payloads and use the response results to further exploit/explore vulnerabilities)
Real examples: PentestGPT
Large Scale Parallelized Social Engineering
For attacks against larger and security-adept companies, social engineering is a more viable option for entry. While criminals are already using mass AI-generated phishing emails, these can be caught (I’m assuming) by savvy end users or existing security solutions. Sophisticated multi-turn hyper-personalized exploits, like those used by North Korean actors on security researchers, are much harder to spot but also require significant investment by the attackers.
In efforts to scale up and increase the chances of exploitation, the Softmax Syndicate uses a complex network of bots that are personalized to specific individuals.
Stages:
Access a contact list of employees (scrape LinkedIn)
For each employee spawn a personalized LLM-based agent which begins with passive information gathering and over time begins active exploitation attempts.
Each agent does a deep dive into the background of an individual using pre-existing OSINT tools and past breach databases
They then craft a set of social media profiles around a personality most likely to establish contact (either impersonating a known individual or synthesizing a personality with similar interests). For a period of time, these profiles generate content and establish relationships on social media to build credibility.
The agent then reaches out to the unsuspecting employee using a novel attack vector personalized to them (Instagram DMs, cold call email, linked-in question, GitHub issue). The agent may choose to establish a history of benign contact before sending an exploit payload, making it much harder to detect.
If the agent fails, a new one is spawned and a new set of accounts are created to repeat the exploit attempt.
This of course won’t have a 100% penetration rate, but odds are for a large enough organization, there will be at least a few individuals with sufficient internal access to compromise the victim company.
Real examples: LLM OSINT (WSJ demo)
Polymorphic Malware
While ChatGPT can be asked to write code — and malware is just code for a specific malicious purpose — the Softmax Syndicate takes this to the next level with adaptive malware.
For this, they use 3 layers of adaptation in their malicious payloads:
Byte-code adaptation. Malware detection traditionally works using signature matching with a database of known exploits (e.g. YARA rules) and by analyzing suspicious behavior (e.g. attempting to edit system files). Using custom coding models, the group is able to synthesize completely novel executables for every use/victim/attack. Knowledge of the target company and its security tools allows for steering the code-scrambling model to generate payloads that avoid interacting with the system in a way that could increase its likelihood of detection.
System adaptation. Once on the system, the malware can adapt and exfiltrate confidential information. Traditional payloads may attempt to upload a hard-coded list of confidential files (
/etc/passwd) but this more sophisticated payload could scan the file system to extract user-specific files and credentials. Analysis of system network traffic (ports and protocols) could allow the malware to choose the more common and least detectable method for transferring data back to the attackers.User adaptation. Using an embedded lightweight model, the malware can also adapt to the user behavior on the infected system. Using a list of software running on the computer (e.g. slack) the malware could also trick the user into escalating permissions by renaming itself on the file system (e.g.
slack-updater.sh) and in the process list. This would also mean that even with the knowledge that malware was on a set of systems, a security team would potentially have no idea what it might be called, where it’s stored, or any sort of fixed signature (assuming #1 was enough also to heavily mask any shared identifiable characteristics) between the instances.
Real examples: I was unable to find an actual example of a tool like this although several blogs have demoed malware obfuscation with ChatGPT.
Rapid Dynamic Lateral Exploitation
Once the group has sufficiently rooted itself into the systems of the victim organization, it can grow its reach, maintain access, and potentially jump to other organizations with complex lateral exploits.
Within seconds of gaining access, the syndicate’s AI system can rapidly determine common external and intra-organization contact patterns, tools, and employee language which when fed into a local model or a remote server can be used to impersonate employees and synthesize lateral attacks.
Examples of this could be:
Identifying a pattern of two software engineers sharing links to development executables and within their chat session sending a malicious payload using the same style as their previous communications
Contacting the organization’s clients with malicious payloads or synthetic vendor compromise attacks after identifying the employees with pre-existing customer email threads
Editing internal documentation pages to inject malicious payloads into onboarding guides from a compromised employee’s account
Impersonating the security team, letting everyone know a security breach has been resolved prematurely, or asking everyone to immediately install a malicious “security” update
Real examples: I was unable to find an actual example of a tool like this although this is a potential extension of Deepfakes for a similar purpose.
Generative Chaos
In the event that the syndicate, while monitoring internal communications, begins to believe they’ve been detected, they use generative AI to inject chaos into the organization to attempt to buy time on the victim’s systems.
By scraping internal communications (e.g. via Slack or email threads) the group is able to identify security team members that pose a threat. From compromised accounts, they then perform an internal communications-based DDoS attack on these employees (or really any employees) via all accessible forms of communication. Unlike in a traditional DDoS, these messages would be potentially indistinguishable from normal communication apart from being such high volume. This doesn’t make it impossible for a security team to make progress but it would definitely complicate the process and bring the rest of the organization to a standstill.
Real examples: I was unable to find an actual example of a tool like this but it’s similar to the existing use of LLMs for misinformation campaigns and spam.
Automated Attack Orchestration
What makes the Softmax Syndicate most dangerous? They can scale these attacks without any manual effort. Using a custom-built set of AutoGPT-like agents, they can simply pick an organization and let the system do the rest from spinning up the social engineering campaigns to dynamically generating malware to determining personalized lateral strategies for an organization.
What was once a spooky hacker man running 3 monitor Kali Linux:
Is now just a chain of prompts running humanless on a cluster of cloud GPUs:
Real examples: Not designed explicitly for cyberattacks, AutoGPT (from which ChaoGPT is derived) is an example of constructing LLM-based agents for complex tasks.
Preventions
These examples are meant to be somewhat far-fetched hypothetical scenarios (although as someone who has spent a lot of time with GPT4, I actually don’t think they are that crazy), but that doesn’t mean some form of AI-driven cyberattacks are not possible in the near to medium term.
I can’t confidently say how one could protect against these especially as we haven’t (as far as we know) seen one of these attacks but here are some potential mitigations:
Simulate to the best of your ability a controlled attack like this against your own organization. Use the experience to develop playbooks and educate employees on how to best handle these complex situations.
Invest in AI-based cyber security tools. It could be that the only way to respond is with fast adaptive AI-based defensive tools of your own. (FYI I work for a company that does this so I’m a little biased)
The unpractical but effective solution could be to lock everything down. Air gap your systems, limit digital communications and impose dramatic restrictions on the software and internet traffic allowed in the organization.