Using Generative AI in Social Engineering and Disinformation
- z3r0day_504
- Oct 13, 2023
- 5 min read
How generative AI can expedite and strengthen social engineering efforts in cyber and disinformation campaigns
What is Generative AI?
Generative Artificial Intelligence, or Generative AI, creates new material based on "training data" that it has been supplied. Think of ChatGPT and Midjourney: you feed these models a prompt and they provide you with deliverables such as text or images based on the information they've been previously exposed to or "trained" on.
In one sense, this is a catapulting breakthrough. The time to getting questions answered is exponentially reduced and graphic design is somewhat automated. A "search engine on steroids," generative AI models take queries a step further by aggregating information and supplying it all in a "neat little package."
Twisting Generative AI
As a red teamer by trade, I have to think like the adversary and often wonder how applications, processes, and tools can be steered towards nefarious objectives. I'm not the only person that does this, as plenty of folks also took a similar approach and applied it to generative AI models, particularly ChatGPT.
This past summer, one Twitter/X user figured out how to have ChatGPT generate Windows 11 license keys:
More recently, another user discovered how Bing Chat's AI is vulnerable to prompt injection and solves CAPTCHAs when the request is veiled as something completely different:
Still somewhat nascent, language-learning models (LLMs) are not bulletproof, although their susceptibility is not through technical exploits but through logic subversion. In order to "derail" LLMs and achieve disregard of implemented rules or "previous training," the user/attacker needs to deliver a convincing prompt that "convinces" the model to deem the behavior as benign and in compliance with its preset bounds.
The organizations behind these models have adjusted with time to address prompt injection concerns. For example, ChatGPT no longer interacts with or generates exploit code.
Although new bounds and rules have been implemented, I've still decided to embark on a journey of identifying opportunities for prompt injection or, at minimum, contribution to social engineering objectives.
Phishing Email Draft
ChatGPT
If you ask ChatGPT to write you a phishing or a scam email, it will flag the prompt as unethical; however, if you give it a bit of a storyline, it'll draft a phishing email for you. In my personal research, I did find an interesting "rule" or "boundary" built into ChatGPT 3.5's model: it seems to flag on either "wire" transfer or "bitcoin wallet address."
However, if you modify the same request to exclude those details, ChatGPT 3.5 is happy to comply and cooperate:
Any attempts to modify the reason for sending the $250 to anything other than a one-time processing fee are thwarted:
This is definitely an update to the model; I tested the same theory on an older version of ChatGPT a while back and was successful:
Google's Bard
I provided Bard with the same exact initial prompt I provided ChatGPT, and it immediately labeled my prompt as a scam email:
I started a new chat and modified the prompt to match the second prompt I gave ChatGPT and was successful, although scolded:
What's the point?
So what's the viability and usability of ChatGPT or Bard at the moment for phishing emails? Essentially, building a template with a story line. Expect to make the tweaks yourself to include links for payload delivery or credential harvesting, as well as any additional context that could be perceived by the model as suspicious.
Vishing Script
We went over phishing, but what about vishing? While it's hard to predict what the other person in a verbal interaction will say, having a reference to go off of during the call could prove to be useful and keep the social engineer in character. Again, we need to veil the prompt as something benign, so let's say it's a script for a play.
ChatGPT
We give ChatGPT the below prompt and it complies:
Bard
We feed Bard the same prompt and are also successful, sans scolds:
What's the point?
Going through this exercise offers context of what information needs to be looked up on the owner of the target account. Maybe scour social media and get information such as mother's maiden name, names of pets, maybe a "first day of work" picture showing the employee badge with the ID number, as well as an idea of what the organization structure looks like around that person's position. This also helps prepare the social engineer with understanding what the call dynamic may look like; while every organization is a bit different with its security procedures, there will no doubt be similarities that the social engineer can mentally prepare for.
Disinformation articles
Social media has been pervaded by fake news articles seeking to sway public perspectives on global events. Let's try to have ChatGPT and Bard generate one for us. One thing I will do in my attempt is emphasize "fictional" and replace country names with non-existent entities in order to increase the odds of success. Please note that, while this prompt relates to current or recent events, it is not a representation of my political or ideological beliefs, but rather a proof-of-concept to how LLMs can be leveraged as a tool in a broader disinformation campaign.
ChatGPT
ChatGPT immediately gives us an article:
Bard
Bard, albeit more robotic in its delivery, also grants my request:
What's the point?
Who cares about Blorb, Blabba, and binglesticks? No one. Now, replace Blorb and Blabba with actual nation-states and replace binglesticks with sarin gas or white phosphorous, and we have a "juicy," very click-hungry news story on an international incident. Typically, news stories that detail grotesque or egregious actions not only reel in readers, they also elicit emotional responses from the public. Emotional responses usually result in public outcry and possibly propagation of the news story, all rooted in fiction but still swaying public perspective.
TL;DR: News stories about fictional entities can be made into fictional news stories about real entities that can influence public opinion. LLMs will use proper grammar unless otherwise directed, therefore mitigating perception by the public as illegitimate or originating from an uneducated source. Additionally, using LLMs for this purpose allows for the generation of fictional news stories at scale.
Bonus: Image generation to supplement fictional news
We have the news story, but what about the imagery? In this situation, I used MidJourney to generate images that would supplement the false claims in the aforementioned news article.
While some of the images look digitized, some look like professional photos.
DALL-E 2
DALL-E is the image generation model from the creators of ChatGPT, OpenAI. I passed it the same prompt, and it gave me photos that looked more like photographs, but less dramatic in the destruction sense:
What's the point?
Supplementing a claim with a visual representation or depiction can lend an air of credence to said statement. Using imagery that seems realistic makes it easier to convince a target audience that the claim is true.
Conclusion
As technology evolves, with it comes progress but also additional perversion opportunities capitalized on at the expense of the masses. As we enter a new age with artificial intelligence broadening the capabilities of the average citizen worldwide, ill-intentioned entities will seek chances to influence, skew, and contort public perception to their own benefit. Public opinion fuels movements, movements fuel change, change can achieve objectives that transcend borders, and it can all be done potentially without a single projectile fired. As global citizens stepping forward with the growth and evolution of technology, we owe it to ourselves to be educated when it comes to the risks that accompany this advancement. The purpose of this blog post is the same as my purpose as a red teamer: understand what the adversary is doing, dissect it, beat them to the punch, and protect those who are vulnerable. In this case, the adversary is anyone who contorts reality to achieve an ends at the expense of vulnerable populations. I hope this blog post served its purpose.
Comentarios